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ABSTRACT 


It  is  generally  accepted  that  cost  growth  in  federal  major  defense  acquisition  programs 
(MDAPs)  is  partially  attributable  to  inaccurate  cost  estimates.  Cost  analysts  exhaustively 
analyze  manpower  and  resources  to  provide  accurate  estimates.  However,  the  influence 
of  transaction  costs  is  often  ignored  in  traditional  cost  estimates.  This  thesis  investigates 
the  association  between  cost  growth  and  transaction  costs,  the  real  cost  of  business 
negotiations  and  program  management.  We  collect  MDAP  cost  threshold  breach  data 
and  cross  reference  it  with  a  proxy  for  MDAP  transaction  costs  (Systems 
Engineering/Program  Management  Costs)  to  determine  whether  a  correlation  exists.  We 
use  multiple  logistic  regression  models  to  analyze  the  binary  outcome  of  breach  or  no 
breach.  The  results  show  that  for  MDAPs  with  cost-plus  contracts  there  is  a  statistically 
significant  relation  between  the  likelihood  of  a  cost  threshold  breach  occurring  and  the 
relative  magnitude  of  the  MDAP’s  transaction  costs;  no  such  relation  exists  for  fixed 
price  contracts.  Although  these  results  show  an  association  between  cost  threshold 
breaches  and  transaction  costs,  there  is  no  evidence  of  causality  between  these  two 
variables  and  our  exploration  of  causality  is  a  topic  for  future  research. 
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EXECUTIVE  SUMMARY 


The  Department  of  Defense  (DoD)  2012  major  defense  acquisition  program  (MDAP) 
portfolio  is  estimated  at  $1.6  trillion,  and  contains  86  MDAPs.  While  this  is  a  reduction 
in  the  total  cost  and  number  of  MDAPs  compared  against  2011,  the  total  cost  of  these 
programs  has  increased  by  more  than  $400B  since  their  first  full  cost  estimate 
(Government  Accountability  Office  2013,  6).  Given  the  large  scale  of  spending,  life  cycle 
cost  estimates  for  MDAPs,  which  include  the  acquisition  cost  estimates,  should  be 
credible  information  for  budgetary  and  policy  decisions. 

DoD  decision  makers  use  three  separate  cost  estimates:  from  the  program  office, 
from  a  DoD  component,  and  an  independent  estimate  performed  by  Office  of  Secretary 
of  Defense  Cost  Analysis  and  Perfonnance  Evaluation  (OSD  CAPE).  Despite  these 
efforts,  the  number  of  MDAPs  that  sustain  cost  breaches  (cost  overruns  of  10%  or 
greater)  suggests  that  acquisition  costs  are  frequently  underestimated. 

Transaction  cost,  the  real  cost  of  business  negotiations  and  program  management, 
is  a  component  of  cost  estimates  that  is  often  overlooked  when  estimating  costs  of  a 
program  or  system.  This  research  explores  the  possibility  that  transaction  costs  are  one 
possible  cause  for  low  cost  estimates  of  MDAPs  and  the  resulting  cost  breaches.  This 
thesis  seeks  to  answer  the  following  questions: 

Is  there  a  relation  between  MDAP  transaction  costs  and  cost  breaches? 

Are  cost  breaches  related  to  the  amount,  or  percentage  of  total  costs,  spent  on 
MDAP  transaction  costs? 

By  using  a  proxy  for  transaction  costs  (systems  engineering  and  program 
management  [SE/PM]  costs),  this  thesis  attempts  to  measure  the  relationship  between 
transaction  costs  and  cost  breaches.  While  most  accounting  systems  do  not  capture 
transaction  costs  as  a  separate  category,  the  SE/PM  category  would  certainly  include  such 
costs.  It  is  logical  to  suppose  that  programs  that  are  more  complex  or  have  technical 
difficulties  are  more  likely  to  experience  higher  transaction  costs  which  may  not  have 
been  anticipated  in  the  original  cost  estimate.  Consequently,  such  programs  would  be 
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more  likely  to  see  cost  breaches.  This  research  analyzes  the  level  of  SE/PM  expenditures 
in  major  defense  acquisition  programs  to  determine  whether  a  relationship  exists  between 
transaction  costs  and  cost  breaches.  A  brief  discussion  about  the  data  collection  process 
and  three  models  used  to  answer  the  proposed  thesis  questions  follows. 

The  research  was  done  in  two  parts:  the  collection  of  MDAP  cost  infonnation  to 
build  a  database;  and  the  selection  of  an  appropriate  model  to  analyze  the  data.  The 
database  paired  program  breaches  and  SE/PM  costs.  Program  cost  breach  infonnation 
found  in  the  MDAP  selected  acquisition  report  was  coupled  with  SE/PM  cost  ratios  that 
were  calculated  using  data  obtained  from  MDAP  cost  summary  data  reports.  For 
purposes  of  this  research,  a  cost  breach  is  defined  as  a  cost  overrun  of  10%  or  greater. 

Recognizing  that  transaction  costs  may  not  be  the  only  factor  associated  with  cost 
breaches,  data  were  collected  for  other  factors  such  as  contract  type  and  “maturity” — 
defined  in  this  report  as  the  time  in  years  since  program  initiation.  While  there  are  many 
more  variables  that  may  be  related  to  cost  breaches,  contract  type  and  maturity  were 
selected  based  on  the  availability  of  relevant  data  from  the  two  databases  used,  Defense 
Acquisition  Management  Information  Retrieval  (DAMIR)  and  Defense  Cost  and 
Resource  Center  (DCARC).  Ideally,  the  development  of  a  multivariate  model  that  has 
predictive  power  for  MDAP  cost  breaches  could  serve  as  a  useful  tool. 

Logistic  regression  is  designed  for  use  with  binary-outcome  dependent  variable 
models  and  was  therefore  the  logical  choice  to  examine  the  occurrence  of  program 
breaches.  Logistic  regression  models  use  maximum  likelihood  estimators,  or  logits, 
which  are  the  natural  logarithm  of  the  odds  ratio  for  a  particular  outcome.  Fixed  effects 
logistic  regression  and  population  averaged  logistic  regression  were  both  considered. 
Fixed  effects  logistic  regression  looks  within  programs  for  changes  in  the  EAC  SE/PM 
cost  ratio  and  is  a  useful  tool  for  describing  the  impact  of  short  run  changes  in  the 
MDAP.  In  the  end,  the  population  averaged  logit  model  was  selected  because  it  looks 
across  programs  to  measure  changes  in  the  SE/PM  cost  ratio  and  is  most  useful  in 
measuring  long  run  effects. 
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In  the  first  model,  MDAP  cost  breaches  were  compared  against  estimate  at 
completion  (EAC)  SE/PM  cost  ratios  and  analyzed  using  logit  fixed  effects  regression 
and  logit  population  averaged  regression.  This  analysis  tested  the  hypothesis  that  the 
probability  of  an  MDAP  sustaining  a  cost  breach  is  related  to  its  EAC  SE/PM  cost  ratio. 
The  logit  population  averaged  regression  model  indicates  a  positive  relationship  that  is 
significant  at  the  5%  (p  =  0.05)  level  between  the  likelihood  of  a  cost  breach  occurring 
and  EAC  SE/PM  cost  ratio.  The  salient  information  that  the  logit  population  averaged 
model  can  provide  includes  the  average  marginal  effects  and  marginal  effects  values 
which  describe  the  effect  of  the  independent  variables  on  the  dependent  variable.  For 
instance,  based  on  across  program  observations  the  logit  population  averaged  model 
shows  that  the  average  marginal  effect  of  the  independent  variable  on  the  dependent 
variable  is  0.80.  This  means  that,  on  average,  for  a  1%  increase  in  the  SE/PM  cost  ratio 
there  is  a  corresponding  0.80%  increase  in  the  likelihood  that  a  cost  breach  will  occur. 

In  the  second  model  an  explanatory  variable  was  added:  program  maturity, 
considered  in  this  report  to  be  the  length  of  time  (in  years)  since  the  program  entered 
Milestone  B.  The  rationale  to  consider  the  effect  of  program  maturity  against  cost 
breaches  is  that  older  programs  are  expected  to  have  less  uncertainty  and  risk  than 
younger  programs.  This  model  tested  the  null  hypothesis  that  there  is  no  relation  of  cost 
breaches  with  EAC  SE/PM  cost  ratio  and  program  maturity  as  defined  in  this  report.  The 
model  was  analyzed  using  logit  fixed  effects  regression  and  logit  population  averaged 
regression.  While  program  maturity  alone  did  not  yield  statistically  significant  results, 
there  is  evidence  that  the  maturity  of  a  program  affects  the  probability  of  a  cost  breach 
occurring  in  the  model.  The  average  marginal  effect  of  maturity  in  the  population 
averaged  logit  model  is  0.70  and  for  EAC  SE/PM  cost  ratio  the  average  marginal  effect  is 
0.89. 1  This  means  that,  on  average,  a  one  unit  (1%)  change  in  the  maturity  of  an  MDAP 
equates  to  a  0.70%  change  in  the  likelihood  of  a  cost  breach  occurring.  Similarly,  a  one 
unit  (1%)  change  in  the  EAC  SE/PM  cost  ratio  will  result  in  a  0.89%  change,  on  average, 
in  the  likelihood  of  an  MDAP  incurring  a  cost  breach. 

1  The  statistically  significant  results  that  produced  a  marginal  effect  on  the  dependent  variable  were 
calculated  using  the  natural  logarithm  of  maturity,  as  explained  in  Chapter  IV. 
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The  third  model  introduced  an  additional  explanatory  variable  to  the  previous 
model:  contract  type.  This  model  compared  MDAP  cost  breaches  to  EAC  SE/PM  cost 
ratios,  program  maturity  and  contract  type.  The  rationale  for  considering  contract  type  is 
that  acquisitions  professionals  generally  accept  the  notion  that  firm  fixed  price  contracts 
are  frequently  negotiated  between  the  government  and  contractors  for  less  risky  programs 
and  cost-plus  contracts  are  negotiated  for  projects  which  have  greater  uncertainty  and  are 
perceived  to  be  more  risky  (Government  Accountability  Office  2009b).  Logic  suggests 
that  riskier  MDAPs  should  have  cost-plus  contracts  and  experience  more  cost  breaches 
than  their  less  risky  counterparts.  This  last  model  tests  the  null  hypothesis  that  there  is  no 
relation  between  cost  breaches  and  EAC  SE/PM  cost  ratio,  program  maturity  and  MDAP 
contract  type.  Only  cost-plus  contracts  were  observed  to  have  statistically  significant 
results,  at  the  10%  level.  For  cost-plus  contracts,  including  the  maturity  of  a  program  has 
an  effect  on  the  probability  of  a  cost  breach  occurring.  According  to  the  population 
averaged  model,  the  average  marginal  effect  of  maturity  is  0.21  and  EAC  SE/PM  cost 
ratio  is  1.00.  This  means  that,  on  average,  for  a  one  unit  (1%)  change  in  program  maturity 
the  likelihood  of  a  cost  breach  occurring  will  change  by  0.21%.  Moreover,  for  a  one  unit 
(1%)  change  in  the  EAC  SE/PM  cost  ratio  the  likelihood  of  a  cost  breach  occurring  will 
change  by  1.00%,  on  average.  The  results  of  this  last  model  show  that  for  cost-plus 
contracts  as  the  EAC  SE/PM  cost  ratio  increases  there  is  a  corresponding  increase  in  the 
probability  of  a  cost  breach  occurring.  Although  there  is  correlation  between  SE/PM  cost 
ratio  and  cost  breaches,  it  cannot  be  assumed  that  transaction  costs  are  a  causal  factor  for 
cost  breaches. 

In  this  research,  the  model  which  included  the  explanatory  variables  maturity  and 
cost-plus  contract  types  yielded  statistically  significant  results  and  had  the  largest  values 
for  the  average  marginal  effects  of  EAC  SE/PM  cost  ratio.  These  contracts  exhibited 
relationships  between  (1)  the  EAC  SE/PM  cost  ratio  and  the  likelihood  of  a  cost  breach 
occurring;  and  (2)  program  maturity  and  the  likelihood  of  a  cost  breach  occurring.  Future 
work  to  better  understand  the  relationship  between  transaction  costs  and  cost  estimation 
may  require  the  development  of  a  more  complex  model.  Hopefully,  this  thesis  will 
provide  a  useful  foundation  toward  building  a  better  model. 
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I.  INTRODUCTION 


The  acquisition  process  is  the  means  by  which  the  federal  government  acquires 
supplies  or  services  by  contract.  For  the  Department  of  Defense  (DoD),  the  acquisition 
process  is  used  to  provide  support  and  services  for  military  operations.  The  sequential 
three-step  process  for  acquiring  supplies  and  services  is:  (1)  the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS),  which  identifies  system  requirements;  (2) 
the  Planning,  Programming,  Budgeting  and  Execution  (PPBE)  process,  which  allocates 
resources;  and  (3)  the  acquisition  process,  which  obtains  the  supplies  or  services  that 
satisfy  defense  needs  (Schwartz  2013,  3).  This  acquisition  process  includes  the  “design, 
engineering,  construction,  testing,  deployment,  sustainment,  and  disposal  of  weapons  or 
related  items  purchased  from  a  contractor”  (Acquisition  Central  2013)  and  is  managed  by 
the  program  manager  (PM). 

The  program  manager  is  “the  designated  individual  with  responsibility  for  and 
authority  to  accomplish  program  objectives  for  development,  production,  and  sustainment  to 
meet  the  user’s  operational  needs.”  The  PM  reports  program  cost,  schedule,  and  performance 
information  to  the  milestone  decision  authority  (MDA)  (Department  of  Defense  Instruction 
5000.01  2003). 2  Weapon  system  acquisitions  fall  into  three  categories:  ACAT  I,  II  or  III. 
Major  defense  acquisition  programs  (MDAPs)  ACAT  I  acquisition  programs  are  estimated  to 
have  expenditures  greater  than  $365M  in  research,  development,  technology  and  engineering 
(RDT&E)  or  $2.190B  in  procurement,  including  all  planned  increments,  in  FY  2000  constant 
dollars;  in  addition,  programs  may  be  designated  as  ACAT  I  by  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  (Defense  Acquisition  University  2011b). 
ACAT  I  programs  are  commonly  referred  to  as  major  defense  acquisition  programs 
(MDAPs).  Acquisition  programs  that  do  not  meet  the  requirements  for  ACAT  I  are 
categorized  as  ACAT  II  or  III  programs.  This  thesis  will  focus  on  cost  overruns  for  ACAT  I 
programs. 


2  The  MDA  has  overall  responsibility  for  the  program  and  is  responsible  for  reporting  program  cost, 
schedule  and  performance  information  to  higher  authority,  including  Congress  (Department  of  Defense 
Instruction  5000.01  2003). 
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Program  managers  use  the  systems  engineering  process,  an  overarching  method 
that  transitions  “from  a  stated  capability  to  an  operationally  effective  and  suitable 
system,”  (Defense  Acquisition  University  2011b)  to  effectively  manage  their  programs. 
The  systems  engineering  process  is  applied  throughout  the  entire  life  cycle  of  a  program, 
and  includes  everything  from  system  design  and  development  to  system  operation, 
disposal  and  life  cycle  costing.3  Cost  estimate  inaccuracies  are  identified  as  a  leading 
cause  of  system  cost  growth  (Blanchard  and  Fabrycky  2010,  567).  This  report  seeks  to 
better  understand  cost  overruns  by  looking  at  transaction  costs,  a  significant  component 
of  program  management,  and  their  relation  to  MDAP  cost  threshold  breaches.  In 
particular,  the  research  looks  at  these  questions: 

•  Is  there  a  correlation  between  MDAP  transaction  costs  and  acquisition 
cost  threshold  breaches? 

•  Do  underestimated  transaction  costs  contribute  to  MDAP  cost  threshold 
breaches? 

•  Are  cost  threshold  breaches  related  to  the  amount,  or  percentage  of  total 
costs,  spent  on  MDAP  transaction  costs? 

A.  COST  OVERRUNS  IN  MAJOR  DEFENSE  ACQUISITION  PROGRAMS 

In  this  report,  cost  overruns  are  defined  using  earned  value  management  (EVM) 
terms.  The  DoD  has  used  the  EVM  system  since  1996  to  evaluate  contractor  management 
systems  (Defense  Acquisition  University  2011b).  According  to  EVM,  a  cost  overrun 
situation  occurs  whenever  the  actual  cost  of  the  work  performed  to  complete  the  project 
is  greater  than  the  budgeted  cost  of  the  work  required  to  complete  the  project.  The  extent 
of  the  cost  overrun  can  be  determined  by  evaluating  the  difference  between  the  actual 
cost  and  budgeted  cost.  The  severity  of  a  cost  overrun  may  warrant  certain  actions  to  be 
taken  by  the  program  manager.  EVM  and  the  consequences  of  sustaining  a  cost  overrun 
will  be  discussed  further  in  Chapter  II. 

The  prevalence  of  cost  overruns  in  major  defense  acquisition  programs  suggests 
that  there  may  be  a  component  of  cost  estimations  that  is  not  accurately  considered.  As 
defined  by  the  Defense  Acquisition  University  (DAU),  a  cost  overrun  is  “the  amount  by 

3  For  more  information  on  life  cycle  costing  see  Benjamin  S.  Blanchard  and  Wolter  J.  Fabrycky, 
Systems  Engineering  and  Analysis  5th  Ed.  (Boston:  Prentice  Flail,  2011),  566-631. 
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which  a  contractor  exceeds  the  estimated  cost  and/or  the  final  limitation  (ceiling)  of  the 
contract”  (Defense  Acquisition  University  2011b).  The  Government  Accountability 
Office  (GAO)  reported  that  the  96  active  MDAPs  in  FY  2011  sustained  acquisition  cost 
growth  of  $74. 4B,  collectively.  Furthennore,  about  60%  of  this  cost  growth  can  be 
attributed  to  procurement  costs  (exclusive  of  procurement  quantity  changes)  and  research 
and  development  costs  (United  States  Government  Accountability  Office  2012).  A  2011 
GAO  report  stated  that  “inaccurate  cost  estimates  are  responsible  for  the  strongest 
correlation  with  net  cost  growth  changes  and  are  associated  with  40  percent  of  the 
accumulated  cost  overruns”  (GAO  2011,  2).4  A  2007  RAND  study  on  the  cost  growth  in 
DoD  weapons  systems  analyzed  total  program  and  procurement  cost  growth  for  46 
programs.  The  study  detennined  that  the  cost  growth  ratio  across  all  programs  was  1 .46, 
or  46  percent,  more  than  the  cost  estimate  for  Milestone  B  (program  initiation)  (Obaid 
Younossi  et.  al  2007,  xvi). 

A  recent  study  looked  at  Department  of  Defense  contractors  and  found  that  they 
earn  above-”nonnal”  profits  when  compared  to  non-defense  industry  peers.  It  reports  that 
“cost  growth  occurs  steadily  throughout  the  program  lifespan”  and  goes  on  to  further 
suggest  “that  younger  programs  are  not  perfonning  better  than  older  programs”  (Wang 
and  San  Miguel  2012).  This  corroborates  the  Defense-Industrial  Initiatives  Group,  Center 
for  Strategic  and  International  Studies  findings  in  their  report  entitled  Cost  and  Time 
Overruns  for  Major  Defense  Acquisition  Programs. 

Perhaps  there  are  acquisition  costs  that  may  not  be  completely  reflected  in  life 
cycle  cost  estimations:  transaction  costs.  Is  it  possible  that  underestimated  transaction 
costs  have  been  contributing  to  cost  growth  for  MDAPs  for  decades?  Cost  estimates  that 
better  account  for  transaction  costs  should  be  more  accurate  (relative  to  original 
estimates).  This  thesis  addresses  transaction  costs,  a  component  of  system  cost  that  is  not 
well  considered  in  cost  estimates.  Is  there  a  correlation  between  transaction  costs  and  cost 
estimate  overruns  in  MDAPs?  A  better  understanding  of  transaction  costs  may  help  to 


4  Information  about  improving  cost  estimating  methodology  can  be  found  in  the  GAO  Cost  Estimating 
and  Assessment  Guide,  GAO-09-3SP,  “Best  Practices  for  Developing  and  Managing  Capital  Cost 
Programs.” 
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explain  the  difference  between  actual  and  estimated  costs  in  MDAPs,  which  could  lead  to 
more  realistic  cost  estimates  and  better  forecasting  of  cost  breaches. 

To  measure  transaction  costs,  this  study  uses  a  proxy  measure  that  incorporates 
systems  engineering  and  program  management  costs.  According  to  MIL-STD-881C, 
system  engineering  is  defined  as  “the  technical  and  management  efforts  of  directing  and 
controlling  a  totally  integrated  engineering  effort  of  a  system  or  program”  (Department  of 
Defense  2011,  221).  Program  management  is  defined  as  “the  business  and  administrative 
planning,  organizing,  directing,  coordinating,  controlling,  and  approval  actions 
designated  to  accomplish  overall  program  objectives,  which  are  not  associated  with 
specific  hardware  elements  and  are  not  included  in  systems  engineering”  (Department  of 
Defense  2011,  222).  The  similarities  in  the  definitions  for  systems  engineering  and 
program  management  can  lead  program  managers  and  contractors  to  different 
interpretations  for  allocating  program  costs.  In  either  case,  whether  the  costs  are  allocated 
to  systems  engineering  or  program  management,  it  is  reasonable  to  assume  that  a 
combination  of  the  two  will  provide  an  approximate,  albeit  indirect,  measure  of  the 
transaction  costs  experienced  by  a  program. 

This  study  seeks  to  understand  the  impact  of  transaction  costs  on  MDAP  cost 
overruns.  Specifically,  it  looks  at  significant  cost  overruns,  defined  as  “cost  breaches.”  A 
cost  breach  occurs  when  program  costs  exceed  the  approved  acquisition  program  baseline 
(APB)  by  10%,  or  greater.  Cost  breaches  and  MDAP  APBs  are  discussed  in  further  detail 
in  subsequent  sections  of  this  report.  For  an  MDAP  that  program  managers  feel  may 
incur  a  cost  breach  it  seems  reasonable  to  assume  that  the  SE/PM  costs  may  increase  as 
managers  proactively  respond  to  the  program  manager’s  direction.  Programs  that  have 
recently  experienced  a  cost  breach  may  also  see  increasing  SE/PM  costs  as  a  reaction  to  a 
program  manager’s  decisions  and  direction.  Can  cost  breaches  be  related  to  the  amount  of 
SE/PM  cost,  as  a  percentage  of  total  cost,  in  a  program? 

B.  DEPARTMENT  OF  DEFENSE  (DOD)  COST  ESTIMATION 

The  purpose  of  cost  estimation  in  the  DoD  is  to  provide  a  reasonably  good 
prediction  of  the  costs  associated  with  a  program  through  its  operational  life — that  is,  to 
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estimate  all  the  major  components  of  life  cycle  costs.  There  are  many  different  ways  to 
define  life  cycle  costs,  however  it  is  generally  accepted  that  life  cycle  costs  are  the  sum  of 
the  following  elements:  research  and  development  (R&D)  costs,  production  and 
construction  costs,  operation  and  support  costs,  and  retirement  and  disposal  costs 
(Blanchard  and  Fabrycky  2011).  Acquisition  costs,  which  consist  of  the  R&D  costs  and 
the  production  and  construction  costs,  compose  a  significant  portion  of  a  program’s 
estimated  life  cycle  cost. 

For  the  purpose  of  this  report,  cost  estimation  in  acquisitions  is  considered  to  be 
“the  summation  of  individual  cost  elements,  using  established  methods  and  valid  data,  to 
estimate  the  future  costs  of  a  program,  based  on  what  is  known  today”  (Government 
Accountability  Office  2009a).  In  a  1972  GAO  report  titled  “Theory  and  Practice  of  Cost 
Estimating  for  Major  Acquisitions,”  it  was  noted  that  DoD  cost  estimating  techniques 
across  DoD  programs  were  inconsistent  in  their  level  of  detail  and  that  the  cost  estimating 
techniques  used  by  the  DoD  programs  varied  widely. 

GAO  believed  these  disparities  typically  resulted  in  inaccurate,  low  cost 
estimates.  In  response  to  these  findings  and  in  the  same  1972  GAO  report,  GAO 
published  its  first  version  of  “Basic  Characteristics  of  Credible  Cost  Estimates,”  a 
descriptive  listing  of  basic  characteristics  that  GAO  advises  should  be  found  in  program 
cost  analyses  to  this  day.  Although  this  guidance  has  been  in  existence  for  years,  GAO 
case  studies  provide  evidence  that  in  many  programs  the  guidance  has  not  been  followed. 
Some  of  the  problems  noted  in  that  report  are: 

•  Known  costs  had  been  excluded  without  adequate  or  valid  justification; 

•  Historical  cost  data  used  for  computing  estimates  were  sometimes  invalid, 
unreliable,  or  unrepresentative; 

•  Understanding  the  proper  use  of  the  estimates  was  hindered  because  the 
estimates  were  too  low; 

•  Readily  retrievable  cost  data  that  could  serve  in  computing  cost  estimates 
for  new  weapon  systems  were  generally  lacking;  and 

•  Organized  and  systematic  efforts  were  not  made  to  gather  actual  cost 
information  to  achieve  comparability  between  data  collected  on  various 
weapon  systems  or  to  see  whether  the  cost  data  the  contractors  reported 
were  accurate  and  consistent. 
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In  an  effort  to  help  government  managers  and  auditors  ensure  credible  cost 
estimates,  the  GAO  published  the  “GAO  Cost  Estimating  and  Assessment  Guide:  Best 
Practices  for  Developing  and  Managing  Capital  Program  Costs”  in  2009.  The  majority  of 
the  guide  (Chapters  1-17)  is  used  to  discuss  best  practices  for  obtaining  credible  cost 
estimates  and  the  relation  between  these  estimates  and  earned  value  management  (EVM) 
(Government  Accountability  Office  2009a,  1).  EVM  is  the  system  used  by  DoD  to 
evaluate  contractor  management  systems  (Defense  Acquisition  University  2011b)  and  is 
explained  in  the  next  chapter.  The  latter  portion  of  the  guide  (Chapters  18-20)  provides 
guidance  for  “integrating  cost  estimation,  system  development  oversight,  and  risk 
management”  to  the  program’s  life  cycle  costs  (Govermnent  Accountability  Office 
2009a,  2). 

In  1992,  Office  of  the  Secretary  of  Defense  Cost  Analysis  Improvement  Group 
(OSD  CAIG)  promulgated  the  “DoD  Cost  Analysis  Guidance  and  Procedures”  (DoD 
5000. 4-M)  to  improve  the  reliability  of  DoD  cost  estimates.  The  CAIG  function  is  now 
conducted  by  Director,  Cost  Assessment  and  Program  Evaluation  (D,  CAPE)  within  OSD 
and  is  responsible  for  overseeing  DoD  cost  estimation  policy  and  procedures  and 
providing  independent  life  cycle  cost  estimates  for  MDAPs  (Defense  Acquisition 
University  2011b,  B-54).5  In  addition  to  the  independent  life  cycle  cost  estimate,  DoD 
Instruction  5000.2  and  DoD  5000. 2-M  require  that  a  program  office  estimate  and  DoD 
component  cost  analysis  estimate  also  be  performed  (Department  of  Defense  1992,  8). 
These  three  estimates  are  based  on  the  MDAP  cost  analysis  requirements  description 
(CARD),  a  document  which  provides  an  overview  of  the  “technical  and  programmatic” 
features  of  an  acquisition  program  (Defense  Acquisition  University  2011b,  B-56).  The 
Director  of  CAPE  considers  these  estimates  and  provides  this  information  for 
consultation  to  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
(USDAT&L)  when  USDAT&L  is  the  milestone  decision  authority  (MDA)  (i.e.,  ACAT 
ID  MDAPs)  (Congress  2009,  Section  101).  In  accordance  with  DoDI  5000.02,  Enclosure 
4,  Table  2,  independent  cost  estimates  for  ACAT  ID  programs  must  be  provided  by  OSD 

5  A  responsibility  of  the  Office  of  Cost  Assessment  and  Program  Evaluation  (CAPE)  is  to  provide 
“independent  analytic  advice”  to  the  Secretary  of  Defense  on  the  cost-effectiveness  of  defense  systems 
(Cost  Assessment  and  Program  Evaluation  2013a). 
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CAIG  before  entering  the  Technology  Development  phase  (Milestone  A),  the  Engineering 
and  Manufacturing  Development  phase  (Milestone  B,  or  program  initiation),  the  Production 
and  Deployment  phase  (Milestone  C)  and  the  full-rate  production  decision  review. 

Cost  estimates  are  expected  to  change  and  presumably  become  more  accurate  as 
the  program  matures.  These  refinements  are  largely  due  to  lessening  uncertainty 
surrounding  the  specific  program  requirements  and  less  outstanding  expenditures 
remaining  as  the  program  progresses  to  completion.  In  order  to  ensure  a  realistic  and 
accurate  cost  estimate  it  is  important  that  MDAP  cost  estimates  be  periodically  re¬ 
evaluated  and  updated  to  account  for  requirements  changes,  actual  cost  expenditures 
instead  of  estimated  costs  and  the  latest  version  of  the  program’s  schedule  (Government 
Accountability  Office  2009a,  37).  GAO  states  that  “relying  on  a  standard  process  that 
emphasizes  pinning  down  the  technical  scope  of  the  work,  communicating  the  basis  on 
which  the  estimate  is  built,  identifying  the  quality  of  the  data,  determining  the  level  of 
risk,  and  thoroughly  documenting  the  effort  should  result  in  cost  estimates  that  are 
defensible,  consistent,  and  trustworthy”  (Government  Accountability  Office  2009a,  12). 

C.  TRANSACTION  COSTS 

Transaction  costs  are  the  costs  associated  with  “source  selection,  periodic 
competition  and  renegotiation,  contract  negotiation  and  management,  performance 
measuring  and  monitoring  and  dispute  resolutions”  (Angelis  et  al.  2008).  Transaction 
costs  are  derived  from  the  complexity  and  riskiness  of  the  work  to  be  accomplished. 
Thus,  transaction  costs  represent  a  “cost  of  doing  business”  that  may  not  be  completely 
captured  in  standard  WBS  estimates. 

Traditional  cost  estimation  is  derived  from  the  Work  Breakdown  Structure 
(WBS);  however,  due  to  the  prevalence  of  cost  overruns  and  inaccurate  cost  estimates,  it 
can  be  deduced  that  something  is  missing  from  this  analysis  of  cost  estimation. 
Transaction  costs  may  be  the  delta  between  actual  and  estimated  costs  that  is  difficult  to 
capture  and  assess. 

In  general,  a  program  has  two  types  of  costs:  contract  costs  and  transaction  costs. 
Contract  costs  are  the  costs  of  production  and  are  usually  captured  in  the  WBS. 
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Transaction  costs  can  be  considered  motivation  or  coordination  costs  and  may  not  be 
adequately  captured  in  the  WBS  (Angelis  et  al.  2008). 

Total  Program  Cost  =  Transaction  Costs  +  Contract  Costs; 

Transaction  Cost  =  Coordination  Costs  +  Motivation  Costs; 

Coordination  costs  include  search  and  information  costs;  bargaining  and  decision 
costs;  policing  and  enforcement  costs.  These  costs  are  affected  by  market  competition 
(contestability),  asset  specificity,  and  the  recurrence  or  frequency  of  a  transaction.  Asset 
specificity,  a  manufacturer’s  specialization  in  the  production  of  a  system  or  machine,  may 
result  in  the  government  becoming  reliant  on  a  sole  provider  and  lead  to  opportunistic 
behavior  by  the  manufacturer.  Motivation  costs  are  those  costs  which  promote  productive 
efforts  and  provide  incentives  to  encourage  investment,  as  well  as  those  costs  which  deter 
unproductive  bargaining  and  opportunistic  behavior.  Motivation  costs  are  impacted  by 
the  complexity  of  the  contract,  the  uncertainty  or  amount  of  risk  that  a  contract  presents 
and  the  contract  type.  For  more  information  about  transaction  costs  see  “Applying 
Insights  from  Transaction  Cost  Economics  (TCE)  to  Improve  DoD  Cost  Estimation” 
(Angelis  et  al.  2007). 

For  an  MDAP,  there  are  many  effective  ways  to  reduce  coordination  and 
motivation  costs: 

•  Reduce  complexity 

•  Reduce  uncertainty 

•  Increase  measurement  and  monitoring 

•  Actively  deter  bad  behavior 

•  Mitigate  uncertainty  from  asset  specificity  with  incentives 

•  Increase  contestability 

A  better  understanding  of  transaction  costs  may  contribute  to  improving  cost 
estimates  and  reducing  the  number  of  MDAPs  which  experience  a  cost  breach. 
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D.  ASSESSING  TRANSACTION  COSTS 


Transaction  costs  are  difficult  to  assess  because  they  are  not  easily  identified  and 
seldom  captured  in  the  accounting  records.  Therefore,  a  proxy  must  be  used  to  measure 
the  effect  of  transaction  costs  due  to  the  lack  of  data.  Angelis,  et.al.  have  suggested  the 
use  of  the  SE/PM  ratio  as  such  a  proxy.  The  SE/PM  ratio  is  defined  as  SE/PM  Cost 
divided  by  Total  Program  Cost,  Equation  1.1. 


SE  /  PM  Cost  Ratio 


SE  /  PM  Costs 
Total  Cost 


(1.1) 


The  SE/PM  cost  ratio  seems  an  appropriate  proxy  for  transaction  costs  because 
program  management  costs  and  transaction  costs  are  both  defined,  in  part,  as  costs 
associated  with  contract  management  (i.e.,  cost,  schedule)  and  performance  measuring 
and  monitoring.  Due  to  differences  in  program  accounting  practices  and  varying 
interpretations  for  classifying  costs  as  Systems  Engineering  or  Program  Management,  it 
is  prudent  to  collect  the  sum  total  of  program  SE/PM  costs  to  ensure  that  the  correct  costs 
are  included  in  the  data  that  is  gathered. 

This  study  will  analyze  the  SE/PM  cost  ratio,  as  a  proxy  for  transaction  costs,  of 
MDAPs  and  the  potential  correlation  between  the  SE/PM  ratios  and  cost  breaches. 
Determining  the  nature  of  any  potential  relationship  between  the  SE/PM  cost  ratio  and 
the  number  of  cost  breaches  experienced  by  a  program  would  test  the  hypothesis  that 
programs  with  higher  SE/PM  cost  ratios  will  experience  more  cost  overruns  than 
programs  with  lower  SE/PM  cost  ratios.  It  is  presumed  that  programs  with  higher  SE/PM 
cost  ratios  are  in  riskier  contractual  relationships  and  have  higher  transaction  costs  and 
will  experience  more  cost  overruns  than  those  programs  with  lower  SE/PM  cost  ratios. 


9 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


10 


II.  COST  BREACHES  IN  MAJOR  DEFENSE  ACQUISITIONS 

PROGRAMS 


Reining  in  cost  growth  for  MDAPs  has  been  problematic  to  the  Department  of 
Defense  (DoD)  for  many  years  (Arena  et  al.  2006).  The  establishment  of  Selected 
Acquisition  Reports  (SARs)  in  1967  was  an  early  attempt  to  better  monitor  MDAP 
performance.  SARs  contain  critical  MDAP  performance,  cost  and  schedule  information 
for  use  within  the  DoD  and  the  Congress.6  Cost  threshold  breaches,  by  program 
appropriation  category,  are  quickly  and  easily  identifiable  within  the  SAR.  If  a  program 
cost  threshold  breach  occurs,  a  brief  explanation  of  how  or  why  the  cost  threshold  breach 
occurred  is  also  found  within  the  SAR.  The  default  threshold  value  for  a  cost  breach  to 
occur  is  10%  cost  growth.  The  various  types  of  cost  breaches  and  their  cost  appropriation 
categories  are  explained  later  in  this  chapter. 

In  1981,  Senator  Samuel  Nunn  and  Congressman  David  McCurdy  introduced  the 
Nunn-McCurdy  Amendment  (10  U.S.C.  §  2433  2006)  in  an  effort  to  curb  MDAP  cost 
growth  by  holding  the  Department  of  Defense  publicly  accountable  to  Congress  for  their 
fiscal  management  of  MDAPs.  The  Nunn-McCurdy  Amendment  became  law  with  the 
FY  1983  Department  of  Defense  Authorization  Act,  which  establishes  consequences  for 
MDAPs  which  exceed  cost  thresholds.  These  consequences  include  that  the  Secretary  of 
Defense  provide  Congress  with  program  cost  estimates  using  original  and  current 
requirements,  as  well  as  certification  to  Congress  that  the  unit  costs  are  reasonable  and 
that  the  management  structure  of  the  program  is  adequate  to  control  future  costs 
(Government  Accountability  Office  2009a,  310-3 1 1). 

A.  TYPES  OF  COST  BREACHES  IN  DOD 

This  thesis  focuses  on  six  important  categories  of  appropriations  where  cost 
breaches  often  occur:  acquisition  procurement  unit  cost  (APUC);  program  acquisition 
unit  cost  (PAUC);  procurement;  research,  development,  testing  and  evaluation  (RDT&E); 

6  The  information  found  in  the  SAR  provides  a  “quick  look”  summary  of  the  MDAP’s  ability  to  meet 
cost,  performance  and  schedule  objectives  and  remain  within  the  thresholds  agreed  upon  by  the  Program 
Manager  and  Defense  Acquisition  Executive. 
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military  construction  (MILCON);  and  acquisition-related  operations  and  maintenance 
(O&M).  These  categories  of  appropriations  were  selected  due  to  their  availability  in  the 
Defense  Acquisition  Management  Information  Retrieval  (DAMIR)  database.  APUC  and 
PAUC  were  selected  for  their  relevance  to  cost  overruns  as  defined  by  Nunn-McCurdy 
cost  breaches.  Procurement  and  RDT&E  appropriations  are  often  identified  in  the  SAR  as 
the  appropriation  which  incurred  the  cost  threshold  breach,  thus  providing  a  logical 
explanation  for  their  inclusion  in  the  data  set.  MILCON  and  acquisition-related  O&M 
appropriations,  though  infrequently  identified  as  appropriations  that  breached  a  cost 
threshold,  were  also  included  in  the  data  because  they  are  components  of  the  program 
acquisition  cost. 

The  following  definitions  are  from  the  Defense  Acquisition  University  (DAU) 
Glossary  and  the  DAMIR  SAR  Data  Entry  Instructions. 

1.  Acquisition  Procurement  Unit  Cost  (APUC) 

APUC  is  the  unit  cost  that  equals  the  program  acquisition  cost  (the  sum  of  all 
procurement  funds)  divided  by  the  total  number  of  fully  configured  end  items  to  be 
procured.  For  unit  cost  reporting  and  APB  breach  purposes,  the  APUC  is  calculated  in 
Base-Year  dollars  (DAMIR  2011,  9-11).  Base-year  dollars  are  calculated  by  using  the 
currently  approved  acquisition  program  baseline  (APB).7 

2.  Program  Acquisition  Unit  Cost  (PAUC) 

PAUC  is  the  unit  cost  that  equals  the  total  program  acquisition  cost  (as  previously 
defined)  divided  by  the  program  acquisition  quantity  (i.e.,  the  total  number  of  fully 
configured  end  items).  For  unit  cost  reporting  and  APB  breach  purposes,  the  PAUC  is 
calculated  in  Base- Year  dollars  (DAMIR  2011).  Base-year  dollars  are  calculated  by  using 
the  currently  approved  APB. 

7  The  acquisition  program  baseline  (APB)  is  the  baseline  that  reflects  the  threshold  and  objective 
values  for  the  minimum  number  of  cost,  schedule,  and  performance  attributes  that  describe  the  program 
over  its  life  cycle  (Defense  Acquisition  University  2011b).  In  the  event  of  a  cost,  schedule  or  performance 
objective  breach,  the  APB  can  be  revised;  this  is  called  “rebaselining”  the  MDAP.  The  initial  APB  is 
referred  to  as  the  “original”  baseline  and  for  a  rebaselined  program  the  revised  APB  is  called  the  “current” 
baseline.  More  information  about  the  rebaselining  process  can  be  found  in  Department  of  Defense 
Instruction  5000.02,  Enclosure  4,  Table  6. 
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3. 


Procurement 


Procurement  appropriations  fund  those  acquisition  programs  that  have  been 
approved  for  production  (to  include  Low-Rate  Initial  Production  (LRIP)  of  acquisition 
objective  quantities),  and  all  costs  integral  and  necessary  to  deliver  a  useful  end  item 
intended  for  operational  use  or  inventory  upon  delivery  (Defense  Acquisition 
University  2011b). 

4.  Research,  Development,  Testing  and  Evaluation  (RDT&E) 

RDT&E  appropriations  fund  the  efforts  perfonned  by  contractors  and  government 
activities  required  for  the  Research  and  Development  (R&D)  of  equipment,  material, 
computer  application  software,  and  their  Test  and  Evaluation  (T&E)  including  Initial 
Operational  Test  and  Evaluation  (IOT&E)  and  Live  Fire  Test  and  Evaluation  (LFT&E). 
RDT&E  also  funds  the  operation  of  dedicated  R&D  installation  activities  for  the  conduct 
of  R&D  programs  (Defense  Acquisition  University  2011b). 

5.  Military  Construction  (MILCON) 

MILCON  appropriations  fund  major  projects  such  as  bases,  schools,  missile 
storage  facilities,  maintenance  facilities,  medical/dental  clinics,  libraries,  and  military 
family  housing  (Defense  Acquisition  University  2011b).  Military  construction  costs  will 
include  only  those  projects  that  directly  support  and  are  uniquely  identified  with  the 
subject  program  (DAMIR  2011). 

6.  Acquisition  Operations  and  Maintenance  (O&M) 

Acquisition  O&M  appropriations  fund  expenses  such  as  civilian  salaries,  travel, 
minor  construction  projects,  operating  military  forces,  training  and  education, 
depot  maintenance,  stock  funds,  and  base  operations  support  (Defense  Acquisition 
University  2011b).  Acquisition-related  operation  and  maintenance  costs  may  include 
acquisition  costs  which,  in  special  cases,  have  been  funded  by  operation  and  maintenance 
(DAMIR  2011). 
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B. 


REPORTING  COST  BREACHES  IN  DOD 


As  previously  mentioned,  the  Selected  Acquisition  Report  (SAR)  provides  a 
status  update  of  MDAP  cost,  schedule  and  perfonnance  infonnation  to  Congress.  The 
document  contains  1 8  sections  which  provide  extensive  details  with  which  decision  makers 
can  make  informed  recommendations  for  the  future  of  the  program.  In  1969,  the  SAR  was 
used  to  provide  periodic  updates  to  the  Senate  Anned  Services  Committee.  Beginning  in 
1975,  SARs  were  required  to  be  submitted  to  Congress  in  order  to  keep  Congress  informed 
of  the  status  of  MDAPs  (DAMIR  2011).  In  the  1980s,  more  federal  legislation  was 
introduced  to  increase  visibility  into  DoD  programs  and  control  cost  growth.  More 
discussion  about  the  DAMIR  database  and  SARs  is  contained  in  Appendix  B. 

The  Nunn-McCurdy  amendment  (Title  10  U.S.C.  §  2433  2006)  to  the  Department 
of  Defense  Authorization  Act  of  1982  established  congressional  oversight  of  MDAPs  that 
experienced  cost  growth  above  certain  thresholds.  The  perceived  effectiveness  of  the 
Nunn-McCurdy  amendment  led  to  it  becoming  a  permanent  requirement  since  1983  in 
the  Department  of  Defense  Authorization  Acts  (Blickstein  et  al.  2011,  12).  Nunn- 
McCurdy  legislation  is  the  current  federal  law  used  to  control  MDAP  cost  growth.  There 
have  been  changes  to  the  Nunn-McCurdy  Act  over  the  years,  most  notably  in  FY  2006 
and  FY  2009.  In  2006,  the  original  baseline  estimate  was  established  as  a  threshold  for 
measuring  cost  growth.  This  prevents  a  program  from  rebaselining  to  avoid  a  Nunn- 
McCurdy  cost  threshold  breach.  In  2009,  Congress  endorsed  a  requirement  that  any 
program  with  a  critical  breach  (defined  below)  is  presumed  terminated  unless  that 
program  has  been  certified  by  the  Secretary  of  Defense.  More  information  about  MDAP 
certification  requirements  can  be  found  in  Public  Law  111-23,  “Weapon  Systems 
Acquisition  Reform  Act  of  2009,”  May  22,  2009;  DoDI  5000.02;  and  Defense 
Acquisition  Guidebook,  Chapter  10. 

The  determination  to  use  the  original  baseline  or  current  baseline  cost  estimate  for 
a  program  resides  in  whether  the  MDAP  has  been  officially  rebaselined.  Early  in  the 
acquisition  phase  when  the  “original”  or  baseline  cost  estimate  for  an  MDAP — also 
known  as  the  APB — is  established  there  may  be  a  considerable  amount  of  uncertainty 

about  future  project  requirements.  The  uncertainty  about  various  project  requirements 
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may  result  in  a  program  exceeding  or  “breaching”  its  original  cost  estimate.  A  cost 
threshold  breach  is  considered  to  occur  when  any  program  cost  expenditures  exceed  the 
APB.  For  the  purposes  of  this  report,  a  cost  breach  is  considered  greater  than  or  equal  to 
10%  above  the  APB.  Cost  breaches  that  are  greater  than  or  equal  to  15%  above  the  APB 
are  classified  as  Nunn-McCurdy  cost  breaches  and  have  special  reporting  requirements 
that  are  not  applicable  to  cost  breaches  which  do  not  cross  the  Nunn-McCurdy  threshold. 

The  Nunn-McCurdy  mandate  requires  reporting  of  specified  cost  information  and 
states  that  “a  Nunn-McCurdy  breach  is  any  increase  in  the  Current  Estimate  (CE)  of  the 
Program  Acquisition  Unit  Cost  (PAUC)  objective  or  Average  Procurement  Unit  Cost 
(APUC)  objective  of  15%  or  more  compared  to  the  currently  approved  [acquisition 
program  baseline]  APB,  or  30%  or  more  compared  to  the  original  APB”  (DAMIR  2011). 
Furthermore,  an  increase  of  15%  or  more  against  the  current  APB  objective,  or  30%  or 
more  against  the  original  APB  objective  is  a  “significant”  Nunn-McCurdy  breach,  and  an 
increase  of  25%  or  more  against  the  current  APB  or  an  increase  of  50%  or  more  against 
the  original  APB  or  revised  original  APB  is  a  “critical”  Nunn-McCurdy  breach.  The  type 
of  cost  threshold  breach  and  the  APB  baseline  that  it  is  compared  against  is  shown  in 
Table  1.  The  APB,  established  by  DoD  components  and  approved  by  the  Milestone 
Decision  Authority,  is  the  baseline  cost  estimate. 


APB  Breach 
(RDT&E, 
Procurement, 
MILCON,  O&M) 

Nunn-McCurdy 

“Significant” 

Breach 

Nunn-McCurdy 

“Critical” 

Breach 

Current  Baseline 
Estimate 

10% 

+  15% 

+25% 

Original  Baseline 
Estimate 

N/A 

+30% 

+50% 

Table  1.  Nunn-McCurdy  Cost  Breach  Thresholds 
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Nunn-McCurdy  cost  threshold  breaches,  or  cost  overruns,  are  based  on  original 
cost  estimates  for  PAUC  and  APUC  at  project  completion  and  in  the  case  of  a  program 
which  has  rebaselined,  cost  threshold  breaches  are  also  based  on  the  current  (i.e., 
rebaselined)  cost  estimate  for  PAUC  and  APUC  at  project  completion.  To  better 
understand  cost  overrun  calculations,  Figure  1  illustrates  how  the  DoD  uses  earned  value 
management  (EVM)  to  combine  “government  management  requirements  and  industry 
best  practices  to  ensure  the  total  integration  of  cost,  schedule,  and  work  scope  aspects  of 
contracts”  (Defense  Acquisition  University  2011a,  43-44).  For  more  information  about 
EVM,  view  the  EVM  Implementation  Guide:  http://www.acq.osd.mil/evm/. 

Figure  1  shows  the  total  allocated  budget  of  the  contract,  which  is  the  sum  of  all 
of  the  budgets  for  work  on  a  contract  and  the  management  reserve,  a  budget  which  is 
withheld  by  the  program  manager  for  risk  management  purposes.  The  budgeted  cost  of 
work  performed  (BCWP),  budgeted  cost  of  work  currently  scheduled  (BCWS)  and  actual 
cost  of  work  perfonned  (ACWP)  for  a  contract  are  the  three  curved  lines  in  the  figure. 
The  BCWP  represents  the  sum  total  cost  of  the  work  packages  that  are  open  or  completed 
for  a  specified  amount  of  time  and  the  BCWS  represents  the  sum  total  cost  of  the  work 
that  was  budgeted  for  completion  within  a  specified  amount  of  time.  The  BCWS  line  is 
also  the  contract’s  performance  measurement  baseline  (PMB)  and  represents  the 
contract’s  overall  time -phased  budget  plan.  The  ACWP  is  the  sum  of  actual  costs  that 
have  been  incurred  to  accomplish  the  work  perfonned  within  a  specified  amount  of  time. 
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Figure  1.  Earned  Value  Management  (From  Defense  Acquisition  University  2012) 

The  Estimate  at  Completion  (EAC)  is  the  sum  of  the  ACWP  and  the  estimate  to 
completion  (ETC)  for  the  remaining  work.  The  ETC  can  be  calculated  using  previously 
defined  terms,  the  cost  performance  index  (CPI)  and  the  schedule  performance  index 
(SPI).  The  CPI  is  calculated  by  dividing  BCWP  by  ACWP  and  is  a  measure  of  cost 
efficiency.  The  SPI  is  calculated  by  dividing  BCWP  by  BCWS  and  is  a  measure  of 
schedule  efficiency.  The  formula  for  calculating  ETC  is: 

ETC  =  (BAC -BCWP) /(CPI*  SPI)  (2.1) 

When  the  EAC,  a  cost  estimate  for  the  entire  contract,  is  a  higher  cost  than  the 
Budget  at  Completion  (BAC),  the  established  baseline  cost  estimate  of  the  contract,  a 
cost  overrun  is  projected.  At  “Time  Now”  in  Figure  1  the  contract  is  not  performing  as 
well  as  it  was  predicted  to  perform.  The  cost  for  ACWP  exceeds  the  cost  for  BCWP,  thus 
creating  a  negative  cost  variance  and  driving  the  EAC  above  the  BAC.  Conversely,  if  the 
EAC  is  less  than  the  BAC  a  positive  cost  variance  is  created  and  a  cost  underrun  is 
projected  since  the  contract  is  not  expected  to  expend  the  entire  allotted  budget.  Prior  to 
project  completion,  to  calculate  the  expected  cost  overrun  the  current  cost  estimate  must 
be  revised  to  incorporate  ACWP  costs  and  requirements  modifications  to  the  existing 
contract.  This  revised  EAC  is  then  compared  to  the  BAC,  the  acquisition  program 
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baseline  (APB,  as  explained  earlier  in  this  chapter)  acquisition  cost  estimate  for  the 
contract.  The  exact  percentage  of  the  cost  overrun,  based  on  the  budget  that  was 
established  at  the  APB,  can  be  calculated  using  the  following  equation: 


%  Cost  Overrun  = 


EAC 

BAC 


X100 


100 


(2.2) 


Figure  1  also  illustrates  the  cost  and  schedule  overruns  for  a  contract  or  series  of 
contracts  at  project  completion.  At  “Completion  Date”  the  contract  should  be  complete, 
however  the  schedule  variance  (the  difference  between  BCWP  and  BCWS)  which  often 
accompanies  cost  overruns  causes  the  projected  date  to  slip  to  the  right-most  dashed  line. 
At  this  revised  completion  date,  the  cost  difference  between  ACWP  (no  longer  EAC 
because  work  is  complete)  and  BAC  is  determined  and  this  delta  defines  the  size  of  the 
cost  overrun.  All  of  the  scheduled  work  has  been  performed  so  BCWS  will  equal 
BCWP.8  The  cost  overrun  calculation  can  be  made  using  a  similar  formula: 


%  Cost  Overrun 


ACWP 

BAC 


XI 00 


100 


(2.3) 


Having  considered  the  categories  of  cost  breaches  for  MDAPs,  as  well  as  exactly 
how  cost  breaches  are  calculated,  it  is  necessary  to  understand  where  the  cost  breach 
information  resides  for  DoD  MDAPs.  A  single-source  document  streamlines  the  process 
for  quickly  identifying  cost  threshold  breaches  for  MDAPs:  the  selected  acquisition 
report  (SAR). 


C.  SELECTED  ACQUISITION  REPORTS  (SAR) 

The  Defense  Acquisition  Management  Information  Retrieval  (DAMIR)  system 
provides  electronic  Selected  Acquisition  Reports  (SARs)  in  PDF  format  for  active  and 
inactive  MDAPs.  SARs  contain  threshold  breach  data  and  a  brief  explanation  delineating 
the  cause  of  the  cost  breach.  A  snapshot  of  the  “Threshold  Breaches”  section  (Table  2)  of 


8  In  many  instances,  contracts  exhibit  a  relation  between  cost  and  schedule  performance.  For  example, 
program  managers  may  increase  the  man-hours  allocation  of  a  project  in  order  to  meet  a  schedule  deadline. 
An  increase  in  man-hours  allocation,  and  particularly  if  the  work  effort  qualifies  as  overtime  pay,  may 
cause  project  costs  to  increase. 
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the  SAR  provides  this  information  at  a  glance.  The  SAR  is  a  single  report  which  contains 
details  of  critical  parameters  of  an  MDAP  or  major  automated  information  system  (MAIS), 
to  include:  the  responsible  office,  threshold  breaches,  schedule,  performance,  current 
contracts  and  cost  details.  SARs  are  required  to  be  submitted  on  an  annual  basis  and  in 
accordance  with  the  SAR  Data  Entry  instruction  quarterly  SARs  are  sometimes  required. 


Threshold  Breaches 

APB  Breaches 

Schedule 

□ 

Performance 

r 

Cost  RDT&E 

□ 

Procurement 

c 

MILCON 

D 

Acq  O&M 

□ 

Unit  Cost  PAUC 

D 

APUC 

□ 

Nunn-McCurdy  Breaches 

Current  UCR  Baseline 

PAUC 

None 

APUC 

None 

Original  UCR  Baseline 

PAUC 

None 

APUC 

None 

Table  2.  Threshold  Breaches  Report  from  SAR 


SARs  provide  a  snapshot  of  overall  MDAP  performance  and  not  individual 
contract  performance.  For  SARs  which  list  a  single  contract,  it  is  simple  to  deduce  that 
the  information  reported  in  the  SAR  is  the  product  of  the  listed  contract.  However, 
MDAPs  typically  require  several  contracts  to  be  executed,  often  concurrently,  and  in 
many  instances  multiple  contracts  are  listed  on  a  single  SAR.  The  current  contract  section 
of  the  SAR  only  includes  “the  six  largest,  currently  active  contracts  (excluding 
subcontracts)  that  exceed  $40  million  in  Then-Year  dollars”  (DAMIR  2011).  Given  that 
the  standard  user  has  limited  database  pennissions  and  access,  the  granularity  required  to 
determine  specific  contract  performance  within  an  MDAP  cannot  always  be  achieved. 
This  means  that  for  MDAPs  listing  more  than  one  contract  in  a  SAR,  if  there  is  a 
threshold  breach  it  is  often  difficult  to  determine  which  contract(s)  caused  the  threshold 
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breech.  On  occasion,  however,  contract  notes  and  comments  will  provide  amplifying 
information  about  which  contract(s)  caused  the  threshold  breach. 

To  more  accurately  determine  which  contracts  were  responsible  for  threshold 
breaches,  MDAPs  which  listed  only  one  contract  in  the  “Contracts”  section  of  the  SAR 
were  selected.  To  further  simplify  the  contract  selection  process,  most  programs  which 
featured  contracts  with  “effort  numbers”  were  excluded.  The  difficulty  with  the  effort 
numbers  is  that  they  complicate  contract  correlation  across  databases.  For  one  contract, 
the  effort  number  in  DAMIR  does  not  always  accurately  correspond  to  the  contract  effort 
number  in  the  DCARC  database. 

Per  the  promulgated  guidance,  “SAR  tennination  will  be  considered  when  90%  of 
expected  program  deliveries  or  90%  of  planned  acquisition  expenditures  have  been  made, 
or  when  Selected  Acquisition  Reporting  criteria  are  no  longer  met”  (DAMIR  2011). 
Therefore,  the  active  MDAPs  found  in  DAMIR  are  between  the  beginning  of  the 
Engineering  and  Manufacturing  Development  phase  (Milestone  B)  and  90%  complete. 
Inactive  MDAPs  that  are  complete  or  >90%  complete  are  no  longer  required  to  submit 
SARs.  There  are  many  instances  where  the  final  SAR  for  an  MDAP  indicates  that  the 
program  is  >90%  complete,  but  not  100%  complete.  The  use  of  data  which  may  only  be 
90%  complete  is  presumed  acceptable  based  on  the  following  assumptions: 

•  90%  of  delivered  product  probably  accounts  for  more  than  90%  of 
expenditures. 

•  Experience  indicates  that  after  90%  of  the  budget  has  been  expended,  the 
remaining  10%  of  the  expenditures  have  already  been  committed. 

SARs  provide  relatively  high-level  contract  cost  infonnation.  For  this  reason,  it 
was  decided  that  DAMIR  would  be  used  solely  for  the  threshold  breach  data  contained  in 
the  SARs.  The  program  cost  data  found  in  the  DD  Form  1921  (Cost  Data  Summary 
Report,  CDSR)  provided  in  the  Defense  Cost  and  Resource  Center  (DCARC)  database 
contains  significantly  more  contract  detail.  DCARC,  an  entity  of  the  Office  of  the 
Secretary  of  Defense  Cost  Assessment  and  Program  Evaluation  (OSD  CAPE) 
organization,  collects  current  and  historical  cost  and  resource  data  for  MDAPs  and 
MAISs.  This  cost  and  resource  data  is  used  by  government  cost  analysts  to  develop  cost 

estimates  for  government  programs  (Cost  Assessment  and  Program  Evaluation  2013b). 
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While  it  is  normally  not  possible  to  determine  system  engineering  and  program 
management  costs  from  an  MDAP  SAR,  the  work  breakdown  structure  (WBS)  format  of 
the  CDSR  makes  obtaining  this  infonnation  simple.  The  data  extracted  from  the  MDAP 
SARs  and  included  in  this  report  is  located  in  Appendix  D. 
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III.  SYSTEMS  ENGINEERING  AND  PROGRAM  MANAGEMENT 

(SE/PM)  COST  RATIO 

This  chapter  briefly  explores  transaction  cost  economics  in  order  to  introduce  the 
independent  variable,  the  systems  engineering/program  management  (SE/PM)  cost  ratio. 
After  explaining  how  the  SE/PM  cost  ratio  is  calculated,  the  source  of  the  raw  data  that  is 
needed  to  make  the  calculations,  the  Cost  Data  Summary  Report  (CDSR)  (located  in  the 
Defense  Automated  Cost  Information  System  [DACIMS]),  is  discussed. 

A.  TRANSACTION  COSTS 

Transaction  cost  economics  (TCE)  describes  the  economic  theory  behind  cost 
analysis  performed  by  an  institution  in  making  the  decision  of  whether  or  not  to 
participate  in  a  market  and  ultimately  incur  or  avoid  expenses.  It  is  generally  accepted  in 
the  economics  community  that  there  are  three  primary  categories  of  transaction  costs: 
search  and  information  costs;  bargaining  and  decision  costs;  and  policing  and 
enforcement  costs  (Johnson  2005).  To  illustrate  these  cost  categories,  consider  the  non¬ 
monetary  exchanges  which  occur  during  the  marketing  and  purchase  of  a  home.  The 
prospective  homeowners  are  working  with  a  realtor  to  find  an  acceptable  home  based  on 
its  physical  location,  size  and  other  intrinsic  characteristics  (search  and  information 
costs).  The  prospective  homeowners  will  then  negotiate  between  their  realtor  and  the 
sellers’  realtor  to  determine  a  purchasing  price  for  the  home  (bargaining  and  decision 
costs).  Last,  if  the  home  was  purchased  using  money  that  was  borrowed  from  a  mortgage 
lender,  the  mortgage  company  will  ensure  that  the  buyers  are  upholding  their  end  of  the 
agreement  by  paying  the  monthly  mortgage  bill  (policing  and  enforcement  costs). 
Clearly,  there  is  a  real  (time  and  effort)  cost  associated  with  these  three  types  of 
transaction  costs,  albeit  difficult  to  quantify.  Firms  and  corporations  consider  TCE,  if  not 
by  name  then  by  practice,  to  guide  their  decision  to  outsource  or  vertically  integrate  a 
service  or  product  which  they  receive. 

The  previous  paragraph  establishes  that  transaction  costs  are  associated  with 
“source  selection,  periodic  competition  and  renegotiation,  contract  negotiation  and 
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management,  perfonnance  measuring  and  monitoring  and  dispute  resolutions”  (Angelis 
et  al.  2008).  Although  they  are  not  available  directly  in  standard  databases,  one  way  to 
estimate  them  is  by  using  SE/PM  costs.  (The  importance  of  those  costs  in  any  given 
program  could  then  be  estimated  with  the  fraction  of  total  costs  allocated  to  SE/PM.)  As 
defined  in  MIL-STD-881C,  the  combined  definitions  of  systems  engineering  costs9  and 
program  management  costs10  are  representative  of  transaction  costs  and  therefore  make 
SE/PM  costs  a  logical  proxy  for  transaction  costs. 

The  SE/PM  cost  ratio  for  a  program  (the  ratio  of  SE/PM  costs  divided  by  total 
program  costs  as  shown  in  Equation  3.1),  specifies  the  proportion  of  total  program  costs 
that  are  dedicated  to  managing,  integrating  and  directing  the  program: 


SE  /  PM  Cost  Ratio 


SE/PM  Costs 
Total  Cost 


(3.1) 


This  report  hypothesizes  that  a  higher  SE/PM  cost  ratio  is  predictive  of  future  cost 
threshold  breaches.  The  higher  the  SE/PM  cost  ratio  then  the  more  risky  a  transaction  is 
considered — the  converse  is  also  implied.  Categorizing  a  SE/PM  cost  ratio  as  high  or  low 
is  a  judgment  call.  There  does  not  appear  to  be  a  directed  standard  or  normal-practice 
SE/PM  cost  ratio  definition  across  MDAPs.  One  reason  that  the  SE/PM  cost  ratios  may 
vary  is  due  to  the  type  of  weapon  system.  Another  reason  SE/PM  cost  ratios  differ 
between  MDAPs,  and  sometimes  across  contracts  within  the  same  MDAPs,  is  due  to 
subjective  interpretations  among  contractors  (and  program  managers)  about  the  definition 
of  SE/PM  costs  and  non-unifonn  standards  regarding  which  costs  qualify  to  be 
categorized  as  SE/PM  costs  (Stem,  Boito  and  Younossi  2006). 

The  SE/PM  cost  values  used  in  this  thesis  are  extracted  from  the  WBS  line  item 
values  for  “To  Date”  SE/PM  Cost  and  EAC  SE/PM  cost  which  are  listed  on  the  Cost 
Data  Summary  Report  (CDSR),  DD  Form  1921.  The  “To  Date”  SE/PM  cost  is  the  sum  of 


9  Systems  engineering  costs  are  the  technical  and  management  efforts  of  directing  and  controlling  a 
totally  integrated  engineering  effort  of  a  system  or  program  (Department  of  Defense  2011,  221). 

10  Program  management  costs  are  costs  associated  with  the  business  and  administrative  planning, 
organizing,  directing,  coordinating,  controlling,  and  approval  actions  designated  to  accomplish  overall 
program  objectives,  which  are  not  associated  with  specific  hardware  elements  and  are  not  included  in 
systems  engineering  (Department  of  Defense  2011,  222). 
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recurring  and  non-recurring  costs  for  the  SE/PM  WBS  line  item  as  of  the  date  of  the 
WBS  report.  The  EAC  SE/PM  cost  is  the  projected  SE/PM  cost  at  contract  completion. 
The  SE/PM  costs  are  inclusive  of  the  total  contract  costs  less  the  contractor’s  pro (it/loss 
or  fees.  For  clarity,  a  sample  WBS  with  SE/PM  cost  ratio  calculations  is  provided  in 
Appendix  C. 

We  analyze  SE/PM  costs  over  the  life  of  the  MDAP  for  trends.  These  trends  are 
significant  because  the  year-to-year  increase  or  decrease  in  SE/PM  costs  over  the  life  of 
the  MDAP  may  be  a  proactive  or  reactive  measure  by  MDAP  program  managers  and/or 
contractors  to  avoid  a  threshold  breach.  Perhaps,  an  MDAP  with  a  gradually  increasing  or 
decreasing  SE/PM  cost  ratio  may  be  indicative  of  a  future  cost  threshold  breach  or 
prevention  thereof. 

The  numerator  of  the  SE/PM  cost  ratio  is  the  amount  of  SE/PM  cost  expenditures 
and  the  denominator  is  MDAP  expenditures  or  total  costs.  For  completeness  of  analysis, 
the  “To  Date”  SE/PM  and  total  costs,  as  well  as  the  EAC  SE/PM  and  total  costs  are 
recorded.  These  four  numbers  are  necessary  for  the  “To  Date”  SE/PM  cost  ratio  and  EAC 
SE/PM  cost  ratio  calculations,  respectively.  To  illustrate  this  point,  according  to  our 
analysis  a  common  sense  approach  for  evaluating  the  cost  performance  of  MDAP  Project 
X  would  be  to  observe  its  current  or  “To  Date”  SE/PM  cost  ratio  and  its  predicted  or 
EAC  SE/PM  cost  ratio. 

B.  COST  AND  SOFTWARE  DATA  REPORTING 

The  MDAP  database  found  in  the  Defense  Automated  Cost  Information  System 
(DACIMS)  3.5  Library  and  discussed  further  in  Appendix  C,  electronically  lists  the 
recorded  contracts  for  most  MDAPs.  This  Cost  and  Software  Data  Reporting  (CSDR) 
library  contains  folders  of  active  and  inactive  MDAP  contract  data  sorted  by  weapons 
system  types.  More  specifically,  contract  information  within  each  of  these  folders 
includes:  the  Cost  Data  Summary  Report  (CDSR),  DD  Fonn  1921;  the  Functional  Cost- 
Hour  Report,  DD  Form  1921-1;  and  the  Contract  Work  Breakdown  Structure  (CWBS) 
Index  and  Dictionary  among  other  documents.  From  the  CDSR  that  is  found  in 
DACIMS,  we  can  extract  the  line  item  costs  that  we  need  for  the  SE/PM  cost  ratio 
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calculation.  Within  a  single  contract,  it  is  not  unusual  to  find  cost  data  for  both  the  prime 
vendor  who  retains  the  listed  contract  and  subcontracted  vendors  who  support  the 
contract  via  the  prime  vendor.  To  simplify  the  data  collection  process,  only  the  cost  data 
provided  in  the  prime  contract  by  the  primary  vendor  is  recorded. 

A  complication  to  the  data  collection  process  occurs  when  the  list  of  contracts  for 
an  MDAP  in  the  DACIMS  library  does  not  completely  match  the  list  of  contracts  for  the 
same  MDAP  in  its  SAR  (Selected  Acquisition  Report)  which  is  located  in  the  Defense 
Acquisition  Management  Information  Retrieval  (DAMIR)  database.  For  this  report, 
MDAPs  with  cost  threshold  breaches  are  excluded  from  the  data  set  if  the  listed  contracts 
are  not  available  in  both  databases,  DACIMS  and  DAMIR.  This  is  because  we  need  to 
know  the  SE/PM  cost  ratios  for  the  MDAP  during  the  year  when  the  cost  threshold 
breach  occurred. 

For  an  MDAP  which  reports  no  cost  threshold  breaches,  it  is  not  necessary  for  all 
of  the  listed  contracts  to  be  available  in  both  databases.  The  reason  for  this  is  that  reliance 
on  the  SAR  in  the  DAMIR  database  assures  that  no  cost  breaches  occurred.  Trusting  the 
SAR  allows  recording  of  the  SE/PM  cost  ratios  for  the  contracts  that  are  available  in  the 
DACIMS  database  without  fear  of  excluding  a  contract  SE/PM  cost  ratio  for  a  year  that 
contained  a  cost  threshold  breach — since  the  SAR  confirms  that  no  cost  threshold 
breaches  occurred. 

In  many  instances  the  cost  data  between  DAMIR  and  DACIMS  do  not  match 
exactly.  Cost  data  as  reported  in  the  SAR  is  from  the  perspective  of  the  government 
representative,  or  program  manager.  Cost  data  contained  in  the  CDSR  is  reported  by  the 
contractor.  The  difference  between  government  and  contractor  cost  calculations  may 
account  for  some  of  the  disparity  between  the  two  sets  of  reported  cost  data.  Another 
source  of  disparity  is  that  the  CDSR  and  SAR  are  normally  submitted  according  to 
different  timelines.  To  lessen  accounting  inconsistencies,  all  cost  data  were  retrieved 
from  the  DACIMS  library  (CDSR)  and  the  threshold  breech  data  were  obtained  solely 
from  the  DAMIR  database  (SAR).  Although  the  accounting  figures  in  the  two  databases 
are  different,  it  is  assumed  that  the  differences  are  not  so  significant  that  they  would 
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change  the  status  (breach  or  no  breach)  of  any  MDAP  in  regard  to  its  cost  threshold 
breach  criteria  as  reported  in  the  SAR. 

The  potential  for  inconsistency  in  SE/PM  cost  determination  across  MDAPs 
exists  due  to  the  individual  MDAPs  assigning  (perhaps  implicitly  or  unintentionally) 
idiosyncratic  definitions  for  SE/PM  costs  within  their  programs.  Some  insight  as  to  how 
MDAPs  define  various  contract  costs  is  located  in  the  CWBS  Index  and  Dictionary.  The 
CWBS  dictionary  contains  a  description  of  the  technical  and  cost  data  for  the  CWBS 
elements  within  the  contract.  By  utilizing  the  CWBS  it  is  possible,  albeit  tedious,  to 
ensure  to  a  high  degree  of  confidence  that  the  SE/PM  costs  between  contracts  in  this 
study  are  inclusive  of  the  same  types  of  WBS  elements. 

Contract  data  found  in  the  CDSR  is  given  as  an  initial,  interim  or  final  cost 
summary  and  is  submitted  in  accordance  with  the  contract  CDSR  plan  as  approved  by  the 
Office  of  the  Secretary  of  Defense  Deputy  Director,  Cost  Assessment  (OSD  DDCA). 
This  means  that  the  frequency  of  submission  for  cost  data  documents  across  different 
MDAPs  varies  considerably  and  that  contractor  cost  data  may  not  be  submitted  on  an 
annual  basis.  For  this  report,  to  ensure  that  the  cost  data  contained  in  the  CDSR  for  an 
MDAP  is  relevant  to  the  period  covered  by  the  same  MDAP’s  SAR,  the  reporting  periods 
for  the  two  documents  are  verified  to  be  within  the  same  calendar  year.  For  example,  if 
an  MDAP’s  CDSR  is  dated  July  2010  and  the  associated  SAR  is  dated  December  2010 
then  this  program  and  its  data  are  eligible  for  inclusion  in  the  data  set. 


27 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


28 


IV.  METHODOLOGY  OF  ANALYSIS  FOR  COST  BREACHES 

AND  SE/PM  COST  RATIO 


This  chapter  presents  the  methodology  used  to  test  the  hypothesis  that  the 
likelihood  of  a  major  defense  acquisition  program  (MDAP)  cost  threshold  breach  may  be 
associated  with  the  transaction  costs  of  the  MDAP.  While  the  system  engineering/ 
program  management  (SE/PM)  cost  ratio  is  a  proxy  for  transaction  costs,  interactions 
between  other  factors  which  might  be  related  to  cost  breaches  are  acknowledged  and 
explained  using  an  influence  diagram.  Data  were  collected  for  SE/PM  cost  ratio 
determinations,  as  well  as  for  some  of  the  other  factors  that  were  presented  in  the 
influence  diagram.  This  chapter  presents  a  detailed  discussion  that  describes  how  all  of 
the  data  were  arranged  for  use  in  the  statistic  models.  Accordingly,  an  explanation  of  the 
ordinary  least  squares  (OLS)  and  logistic  regression  models  which  were  used  for  the 
analysis  in  this  report  are  included  in  this  chapter. 

A.  HYPOTHESIS 

Numerous  reports  (e.g.,  Bolten  et  al.  2008)  have  studied  cost  growth  in 
Department  of  Defense  (DoD)  programs,  yet  research  that  seeks  to  establish  a  potential 
correlation  between  the  SE/PM  cost  ratio  of  an  MDAP  and  the  occurrence  of  a  cost 
breach  is  limited  (Angelis  et  al.  2008).  Is  a  high  SE/PM  cost  ratio  associated  with  an 
acquisition  program  baseline  (APB)  or  Nunn-McCurdy  breach?  What  is  considered  a 
high  SE/PM  ratio?  A  recent  RAND  study  established  that  MDAP  SE/PM  costs  vary 
between  programs  depending  on  the  program  type.11  For  instance,  the  average  SE/PM 
costs  for  aircraft  development  programs  are  different  from  the  average  SE/PM  costs  of 
weapons  programs. 

This  report  seeks  to  detennine  the  nature  of  any  potential  relationship  between 
transaction  costs  (using  the  SE/PM  cost  ratio  as  a  proxy)  and  the  likelihood  of  cost 
breaches  experienced  by  a  program.  The  hypothesis  is  that  programs  with  higher  SE/PM 

1 1  SE/PM  cost  is  variable  between  MDAPs  because  program  managers  are  idiosyncratic  with  what  is 
included  in  SE/PM  costs.  There  is  no  generally  accepted  standard  of  what  qualifies  as  SE/PM  costs, 
although  it  is  defined  in  MIL-STD-881C  (October  2011)  (Stem,  Boito  and  Younossi  2006). 
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cost  ratios  are  more  likely  to  experience  cost  breaches  than  programs  with  lower  SE/PM 
cost  ratios.  This  is  based  on  the  assumption  that  higher  SE/PM  cost  ratios  are  related  to 
riskier  contractual  relationships  since  more  time,  effort  and  resources  are  expended  to 
meet  performance  and  schedule  deadlines  when  compared  to  less  risky  contracts. 
Programs  with  higher  transaction  costs  may  experience  more  cost  breaches  as  a  result  of 
those  transaction  costs  not  being  accounted  for  in  the  original  cost  estimate. 

The  influence  diagram  in  Figure  2  describes  the  interactions  between  factors 
which  may  be  associated  with  the  occurrence  of  a  cost  breach.  There  are  many  other 
variables  which  affect  cost  breaches,  but  the  variables  identified  in  this  influence  diagram 
(i.e.,  SE/PM  cost  ratio,  program  maturity  and  contract  type)  were  selected  because  they 
can  be  represented  with  the  data  collected.  Beginning  with  the  maturity  of  the  program, 
the  maturity  variable  has  quotation  marks  because  “maturity”  means  different  things  to 
different  readers.  While  in  this  report  “maturity”  is  defined  as  the  time  in  years  since 
program  initiation  or  Milestone  B,  many  will  think  first  about  technology  readiness  levels 
(TRLs).  Program  managers  and  contractors  work  together  to  manage  away  risk  and 
complexity  as  the  program  ages.  This  process  is  captured  by  Figure  2  and  is  later 
discussed.  Concerning  the  risk  and  complexity  of  the  MDAP,  the  program  managers  and 
contractors  can  only  state  educated  opinions  and  informed  assumptions  to  guide  them  in 
the  negotiations  for  an  appropriate  contract  type.  In  some  cases,  the  risk  associated  with 
the  complexity  of  a  new  technology  may  result  in  a  cost  overrun.  In  other  instances, 
perhaps  transaction  costs  associated  with  managing  the  MDAP  contracts  and  business 
negotiations  contribute  to  cost  overruns  and  cost  breaches.  Because  transaction  costs  are 
difficult  to  quantify,  the  SE/PM  cost  ratio  can  be  used  as  a  proxy  to  help  understand  the 
contribution  of  transactions  costs  to  a  program’s  cost  overruns  and  subsequently  the 
probability  of  the  program  having  a  cost  breach.  Of  course  not  all  cost  overruns  are 
considered  cost  breaches.  Current  estimates  which  exceed  the  estimated  cost  of  a 
program  by  any  amount  are  considered  cost  overrun  situations.  Cost  breaches  result  when 
the  amount  of  the  cost  overrun  exceeds  certain  parameters  defined  by  regulation.  In  this 
thesis,  a  default  value  of  10%  cost  overrun  constitutes  the  cost  breach  threshold.  Finally, 
the  dashed  lines  in  Figure  2  represent  factors  that  one  cannot  observe  or  does  not  know 
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how  to  observe  and  must  be  dealt  with  qualitatively  or  by  using  proxies.  The  solid  lines 
represent  factors  that  can  be  quantitatively  evaluated. 


Figure  2.  Influence  Diagram 


There  are  two  types  of  cost  breaches:  APB  and  Nunn-McCurdy  breaches.  In 
order  for  a  program  to  incur  an  APB  breach  or  Nunn-McCurdy  breach,  currently 
estimated  program  expenditures  must  be  greater  than  the  APB  estimate  at  completion  by 
at  least  10%  or  15%,  respectively.  (Cost  breaches  are  discussed  in  detail  in  Chapter  II, 
Section  B.)  The  influence  diagram  shows  that  not  all  cost  overruns  are  cost  breaches  by 
associating  a  probability  with  APB  or  Nunn-McCurdy  breaches.  The  desire  to  determine 
the  probability  of  a  binary  outcome  (breach  or  no  breach)  occurring  in  MDAPs  led  to  the 
decision  to  use  logistic  regression  (logit)  to  model  and  analyze  the  data. 

This  report  takes  steps  toward  developing  a  logistic  regression  model  which 
program  managers  may  be  able  to  utilize  to  determine  whether  their  MDAP  is  in  jeopardy 
of  incurring  a  cost  threshold  breach.  According  to  this  model,  program  managers  will  be 
able  to  estimate  the  probability  of  their  program  incurring  an  APB  or  Nunn-McCurdy 
breach  based  on  the  SE/PM  cost  ratio  which  their  program  maintains.  Three  models  were 
used  to  assess  the  impact  of  an  increase  in  SE/PM  cost  ratio  on  the  likelihood  of  a 
program  experiencing  an  APB  or  Nunn-McCurdy  breach:  a  pooled  OLS  regression,  a 
fixed  effects  model  and  a  population  averaged  model. 


31 


The  null  hypothesis  of  this  research  states  that  better  incorporation  of  transaction 
costs  into  cost  estimates  will  not  significantly  affect  the  ability  of  an  MDAP  to  operate 
within  its  approved  financial  constraints  as  measured  by  cost  breaches. 

B.  DATA 

The  cost  data  for  32  MDAPs  were  collected  to  detennine  the  presence  of  any 
correlation  between  MDAP  cost  breaches  and  transaction  costs.  The  MDAPs  were  not 
randomly  selected,  but  instead  selected  based  on  the  availability  of  relevant  data  from  the 
two  databases  used.  Defense  Acquisition  Management  Information  Retrieval  (DAMIR) 
and  Defense  Cost  and  Resource  Center  (DCARC).  Cost  threshold  breach  infonnation, 
categorized  by  the  cost  appropriation  that  sustained  the  cost  breach,  was  available  from 
the  Selected  Acquisition  Report  (SAR)  in  the  DAMIR  database.  At  a  minimum,  the 
MDAP  SARs  were  generated  annually  as  a  means  for  the  program  manager  to  provide  a 
status  report  to  Congress.  The  line  item  cost  inputs  for  calculating  the  “to  date”  and  EAC 
SE/PM  cost  ratios  were  obtained  from  the  Cost  Data  Summary  Reports  (CDSRs)  found 
in  the  Defense  Automated  Cost  Information  System  (DACIMS)  library  located  within  the 
DCARC  database.  The  32  MDAPs  researched  represent  the  Departments  of  the  Air 
Force,  Anny,  and  Navy,  plus  Joint  programs. 

The  MDAP  data  were  collected  and  organized  within  a  data  table  by  dependent 
and  independent  variables  across  a  time  series  for  each  MDAP  (see  Appendix  D  for  a  list 
of  the  MDAPs  used  in  this  research).  Any  type  of  cost  breach,  APB  cost  breach  or  Nunn- 
McCurdy  cost  breach,  is  considered  the  binary-outcome  dependent  variable  in  this 
analysis:  cost  breach  or  no  cost  breach.  There  are  three  independent  or  explanatory 
variables  that  were  included  in  the  analysis  for  this  report:  EAC  SE/PM  cost  ratio,  time 
since  program  initiation  and  program  contract  type.  While  the  exact  nature  of  the 
relationship  between  cost  threshold  breaches  and  these  explanatory  variables  is  unknown, 
it  is  reasonable  to  suppose  that  the  explanatory  variables  influence  the  cost  performance 
of  the  MDAPs  in  some  manner. 

This  research  uses  binary,  clustered  and  panel  data.  From  the  sample  provided  in 
Table  3,  the  binary  nature  of  the  data  is  shown  to  pertain  to  the  dependent  variable  “Cost 
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Breach.”  In  simple  terms,  the  program  either  sustained  an  APB  or  Nunn-McCurdy 
breach  during  a  specific  calendar  year  or  it  did  not.  The  clustered  nature  of  the  data 
describes  the  various  categories  of  program  types  that  the  data  represents.  For  example, 
in  Table  3  the  MDAP  A  program  may  be  categorized  as  a  ship  building  program  and  the 
MDAP  B  program  may  be  categorized  as  an  aircraft  development  program.  This  is  of 
particular  importance  because  the  average  SE/PM  cost  ratio  is  variable  between  program 
categories  and  this  grouping  must  be  accounted  for  in  any  statistical  analysis. 


Program 

Program 

Number 

Contract 

Year 

EAC 

SE/PM 

APB 

Cost 

Breach? 
(0  -  no, 

1  -  yes) 

N-M 

Cost 

breach? 
(0  -  no, 

1  -  yes) 

"Maturity" 

(years) 

Contract 

Type 
(0  -  Fixed, 
1  -  Cost) 

MDAP  A 

20 

2005 

0.0441 

0 

0 

5 

1 

MDAP  A 

20 

2006 

0.0318 

0 

0 

6 

1 

MDAP  B 

21 

2007 

0.0144 

0 

0 

11 

0 

MDAP  B 

21 

2009 

0.0641 

1 

0 

13 

0 

MDAP  B 

21 

2010 

0.1626 

0 

0 

14 

0 

MDAP  B 

21 

2011 

0.1342 

1 

1 

15 

0 

MDAP  C 

22 

2000 

0.0292 

1 

0 

7 

0 

Table  3.  Sample  Data  Set 


The  importance  of  accounting  for  clustering  is  noted  in  Logistic  Regression  Using 
the  SAS  System  (Allison  2001,  180): 

•  Ignoring  clustering  and  treating  each  observation  as  independent  may 
cause  the  calculated  standard  errors  to  be  underestimated  and  the  test 
statistics  to  be  overestimated. 

•  Applying  conventional  logit  analysis,  which  will  be  explained  in  Section 
C  of  this  chapter,  to  clustered  data  will  produce  standard  errors  that  may 
be  larger  than  those  standard  errors  produced  by  other  analysis  methods. 

•  Clustered  data  can  be  corrected  for  biases  that  may  occur  in  any 
application  of  binary  regression,  i.e.,  a  particular  logit  analysis  method  can 
correct  for  bias  due  to  omitted  explanatory  variables. 
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Last,  the  dataset  in  Table  3  shows  observations  of  multiple  independent  variables 
that  have  been  collected  for  a  program  over  a  time  series  (measured  in  years)  and  in 
statistics  this  type  of  data  collection  is  known  as  panel  data. 

Program  work  breakdown  structure  (WBS)  line  item  costs  for  SE/PM  were 
obtained  from  cost  data  summary  reports  (CDSR,  DD  Form  1921)  found  in  the  DCARC 
database.  The  SE/PM  cost  ratios  calculated  serve  as  the  proxy  for  transaction  costs.  The 
“to  date”  SE/PM  cost  ratios  were  calculated  by  dividing  the  “to  date”  SE/PM  cost  by  the 
“to  date”  total  cost  of  the  program.  Similarly,  the  estimate  at  completion  (EAC)  SE/PM 
cost  ratios  were  calculated  by  dividing  the  EAC  SE/PM  cost  by  the  EAC  total  cost  of  the 
program.  SE/PM  cost  data  ratio  calculations  were  recorded  by  year  for  each  program,  as 
available.  During  the  development  of  this  report,  it  was  decided  that  “to  date”  SE/PM 
cost  ratios  provided  extraneous  infonnation  and  would  not  be  included  in  data  analysis 
since  MDAP  cost  breaches  are  based  on  EAC  cost  projections  and  not  “to  date”  cost 
expenditures. 

In  this  report,  program  “maturity”  describes  the  time  that  has  elapsed  since 
program  initiation  (or  Milestone  B,  the  entry  point  into  the  Engineering  and 
Manufacturing  Development  phase).  The  concept  of  maturity  may  seem  quite  simple; 
however,  this  definition  also  implies  attainment  of  the  DoD  requirements  for  an  MDAP 
to  accomplish  Milestone  B  approval,  namely  that  the  design  and  technology  utilized  in 
the  project  are  mature.12  Using  the  DoD  Milestone  B  requirements  to  define  maturity 
ensures  that  each  program  analyzed  in  this  report  is  assessed  beginning  with  a 
comparatively  equitable  baseline.  Presumably,  as  time  progresses,  a  project’s  contracts 
should  have  a  smaller  likelihood  of  incurring  a  cost  breach  for  each  successive  year  after 
program  initiation  because  there  is  more  available  information  about  the  project  and  the 
realm  of  the  unknown  is  decreasing.  The  increase  in  system  or  program  information  and 
the  corresponding  reduction  in  uncertainty  are  represented  by  the  uncertainty-information 


12  Milestone  B  approval  authorizes  an  MDAP  to  enter  the  Engineering  and  Manufacturing 
Development  phase  of  the  acquisition  process.  Statutory  requirements  for  MDAPs  to  achieve  Milestone  B 
approval  are  found  in  Title  10  U.S.C.  §  2366b.  These  requirements  include  that  the  program  is  certified  by 
the  milestone  decision  authority  to  be  affordable,  fully  funded  through  the  Future  Years  Defense  Program 
(FYDP)  and  that  the  cost  and  schedule  estimates  are  reasonable. 
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tradeoff  as  a  function  of  program  maturity  as  shown  in  Figure  3  (After  Yoe  2000,  2).  This 
figure  suggests  that  the  greatest  uncertainty  in  a  cost  estimate  occurs  in  the  earlier  stages  of  a 
program  when  there  are  more  future  unknowns  as  compared  to  the  later  stages  of  a  program. 


Figure  3.  Uncertainty  and  Infonnation  as  a  Function  of  Maturity. 

(After  Yoe  2000,  2) 

The  time  since  program  initiation  at  Milestone  B  is  obtained  from  the  Selected 
Acquisition  Report  (SAR)  that  is  promulgated  annually  in  December  by  the  MDAP 
program  office  and  an  extensive  listing  of  the  active  and  inactive  SARs  for  all  DoD 
programs  (Air  Force,  Army,  Joint,  Navy)  can  be  accessed  via  the  Defense  Acquisition 
Management  Information  Retrieval  (DAMIR)  database.  The  time  elapsed  since  Milestone 
B  for  a  contract  may  not  always  accurately  represent  the  amount  of  time  that  an  MDAP 
has  been  in  existence.  For  instance,  the  AIM-9  Sidewinder  first  entered  service  the  1950s 
(Wikipedia  2013),  but  many  of  the  later  variants  of  the  AIM-9  were  developed  using 
contracts  from  more  recent  years,  such  as  the  AIM-9X  program  which  began  in  1996 
(MacKenzie  1997).  In  instances  such  as  the  AIM-9X  program,  the  program  initiation  date 
for  the  specific  variant  contracted  was  utilized  to  determine  the  maturity  of  the  program. 

Program  contract  type  is  obtained  from  the  contracts  (CDSRs)  which  are  found  in 
the  DC  ARC  database.  Programs  are  noted  as  having  either  firm  fixed  price  type  contracts 
or  cost-plus  type  contracts.  It  is  presumed  that  riskier  programs  will  have  cost-plus 
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contracts  since  contractors  will  be  hesitant  to  enter  into  firm  fixed  price  contracts  which 
require  them  to  meticulously  manage  the  costs  of  the  work  in  order  to  earn  a  profit.13 

C.  APPROACH  TO  ANALYSIS 

Given  the  nature  of  the  data,  a  good  starting  point  was  to  consider  a  bivariate 
model  to  explain  cost  breaches  and  transaction  costs.  Using  a  standard  bivariate 
regression,  one  variable  would  be  considered  to  predict  an  outcome  variable  (Acock 
2006,  211).  In  the  case  of  this  research,  a  high  value  for  the  independent  variable  of 
SE/PM  cost  ratio  may  manifest  as  a  possible  ex-ante  indicator  for  a  cost  breach  in  an 
MDAP.  It  has  already  been  established  that  we  are  analyzing  clustered  panel  data  and 
that  the  average  SE/PM  cost  ratios  are  different  between  program  types.  Thus,  a  single 
SE/PM  cost  ratio  value  that  would  indicate  a  cost  breach  is  of  no  use  to  a  program 
manager  since  there  is  no  accounting  for  the  variance  across  program  types.  For  example, 
the  overall  SE/PM  cost  ratio  could  be  a  value  that  is  lower  than  the  average  SE/PM  cost 
ratio  for  programs  that  normally  maintain  a  comparatively  high  SE/PM  cost  ratio. 
Certainly  SE/PM  cost  ratio  alone  does  not  indicate  that  an  MDAP  will  have  a  cost  breach 
(e.g.,  Sullivan  2011).  While  other  factors  that  might  impact  cost  breaches  were  taken  into 
account,  data  clustering  has  not  yet  been  taken  into  consideration. 

A  multiple  regression  model  is  the  next  logical  model  that  may  be  considered.  A 
multiple  regression  makes  sense  because  it  would  incorporate  input  from  many  predictors 
or  independent  variables  to  predict  the  outcome  variable  (Acock  2006,  212).  Again, 
finding  a  single  SE/PM  cost  ratio  value  which  may  indicate  an  increased  likelihood  for  a 
breach  does  not  seem  like  it  would  be  helpful  to  a  program  manager  since  SE/PM  cost 
ratios  variables  between  MDAP  types. 


13  A  cost  plus  contract  is  a  type  of  contract  that  provides  for  payment  to  the  contractor  of  allowable 
costs  incurred  in  the  performance  of  the  contract,  to  the  extent  prescribed  in  the  contract.  This  type  of 
contract  establishes  an  estimate  of  total  cost  for  the  purpose  of  obligating  of  funds  and  establishes  a  ceiling 
that  the  contractor  may  not  exceed  without  prior  approval  of  the  Contracting  Officer  (CO).  A  firm  fixed 
price  contract  provides  for  a  price  that  is  not  subject  to  any  adjustment  on  the  basis  of  the  contractor’s  cost 
experience  in  performing  the  contract.  This  type  of  contract  places  upon  the  contractor  maximum  risk  and 
full  responsibility  for  all  costs  and  resulting  profit  or  loss.  It  provides  maximum  incentive  for  the  contractor 
to  control  costs  and  imposes  a  minimum  administrative  burden  on  the  government  (Defense  Acquisition 
University  2011b). 
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Perhaps  instead  of  finding  a  single  SE/PM  cost  ratio  that  would  presumably  alert 
program  managers  to  the  impending  danger  of  an  APB  or  Nunn-McCurdy  cost  breach,  it 
would  be  more  helpful  to  provide  program  managers  with  an  indication  of  the  risk  of 
incurring  a  cost  breach.  A  ratio,  adjusted  to  be  program-specific,  that  program  managers 
may  reference  for  their  decision  making. 

In  the  case  of  an  MDAP  cost  breach,  the  outcome  variable  is  binary:  breach  or  no 
breach.  Logistic  regression  is  a  type  of  regression  that  was  designed  for  binary-outcome 
dependent  variables  (Acock  2006,  250).  Logistic  regression  predicts  the  probability  of 
binary  outcomes  (breach  or  no  breach).  The  S-curve  seen  in  Figure  4  indicates  the 
relationship  between  the  linear  probability  and  logit  models.  In  order  to  fit  this  model,  a 
logit  or  maximum  likelihood  estimation  is  calculated  for  each  observation. 
Mathematically,  the  logit  value  (Equation  4.1)  is  the  natural  logarithm  of  the  odds  ratio 
for  a  given  event  to  occur  (Acock  2006,  256);  and  the  odds  ratio  is  simply  the  ratio  of  the 
event  occurring  versus  the  ratio  that  the  event  will  not  occur.  Each  point  on  the  logistic 
regression  line  in  Figure  4  represents  a  logit  value  which  describes  the  relationship 
between  the  dependent  and  independent  variable. 

Logit  =  In  (odds  ratio)  (4.1) 

Although  the  probability  of  an  MDAP  crossing  a  cost  threshold  is  not  linearly 
related  to  the  SE/PM  cost  ratio,  the  logit  is  and  therefore  the  predicted  logit  of  a  cost 
breach  occurring  is  the  statistical  equivalent  to  the  predicted  probability  of  a  cost  breach 
occurring  (Acock  2006,  253). 
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Comparing  the  LP  and  Logit  Models 

Y 


Figure  4.  Linear  Probability  versus  Logit  Model 
(From  http://www.appstate.edu/~whiteheadjc/service/logit/logit.gif) 

In  this  study,  the  pooled  OLS  regression  serves  as  the  baseline  model.  The  data  is 
considered  “pooled”  because  it  uses  all  of  the  independent  data  observations. 
Unfortunately,  while  the  binary  outcome  is  limited  to  a  maximum  probability  of  1.0 
(100%)  and  minimum  probability  of  0.0  (0%),  OLS  regression  is  not  constrained  to  these 
limits.  Because  it  uses  linear  estimates,  OLS  regression  may  result  in  predicted 
likelihoods  greater  than  1.0  or  less  than  0.0.  Additionally,  OLS  regression  assumes  that 
each  observation  is  independent,  but  we  already  know  that  this  is  not  the  case  with  the 
MDAP  data:  observations  from  within  the  same  cluster  or  program  are  probably  more 
alike  than  observations  from  another  cluster.  In  other  words,  it  is  expected  that  within 
group  outcomes  are  related  differently  than  the  cross-group  outcomes.  The  assumption 
that  each  observation  is  completely  independent  skews  the  calculated  standard  errors  for 
the  data.  A  solution  to  the  problems  presented  by  the  OLS  regression  model  is  to  employ 
one  of  several  maximum  likelihood  estimators  and  panel  data  techniques. 

The  second  model  tested  was  a  fixed  effects  logistic  regression  model.  The  fixed 
effects  or  conditional  logit  model  adjusts  for  bias  introduced  by  omitted  explanatory 
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variables  within  the  clusters  of  data.  It  controls  for  characteristics  which  may  cause  a  cost 
breach  even  if  those  characteristics  cannot  be  measured.  The  standard  logit  model  is 
presented  in  Equation  4.2  (Allison  2001,  181),  where  Pit  represents  the  probability  that  at 
a  specific  time  an  MDAP  breaches  a  cost  threshold,  P;t  is  the  vector  of  regression 
coefficient  which  represents  the  association  of  cost  breaches  and  the  explanatory  variable 
to  a  unit  change  in  the  independent  variable  (SE/PM  cost  ratio)  and  x  represents  the 
explanatory  variable  for  the  respective  MDAP. 

Log(P„/  1-P„)  =/«„x  (4.2) 

The  fixed  effects  (conditional)  logit  model,  which  accounts  for  dependence  of  the 
observations  within  a  group,  modifies  the  basic  logit  model  (Equation  4.3)  (Allison  2001, 
188).  In  this  equation,  a  represents  a  set  of  fixed  constants  for  each  MDAP  in  the  group. 
This  model  measures  the  impact  of  deviations  by  the  independent  variable  from  the 
group’s  (program’s)  mean.  This  model  can  be  considered  to  describe  the  impact  of  short 
run  changes,  i.e.,  that  a  change  in  the  SE/PM  cost  ratio  for  an  MDAP  will  change  its 
probability  of  breaching  a  cost  threshold. 

Log  (Pit/  1-Pit)  =  a  +  J3ltx  (4.3) 

Finally,  the  relationship  between  SE/PM  cost  ratio  and  cost  breach  was  modeled 
using  a  population  averaged  logit  model.  Contrary  to  the  fixed  effects  method  that  looks 
within  programs  for  changes  in  the  SE/PM  cost  ratio,  the  population  averaged  method 
looks  across  programs  to  measure  changes  in  the  SE/PM  cost  ratio.  Therefore,  using 
Equation  4.3  to  explain  the  population  averaged  logit  model,  a  represents  a  set  of 
averaged  constants  for  MDAPs  across  program  groups.  The  population  averaged  method 
is  most  useful  for  measuring  long  run  effects,  i.e.,  measuring  whether  programs  with  high 
average  SE/PM  cost  ratios  are  more  likely  to  breach. 

D.  SUMMARY 

A  logistic  regression  model,  or  a  maximum  likelihood  estimator  such  as  logit 
which  uses  the  natural  logarithm  of  the  odds  ratio,  is  appropriate  for  this  research  because 
it  was  designed  for  use  with  binary-outcome  dependent  variables.  Additionally,  a  fixed 
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effects  logit  model  will  adjust  for  bias  introduced  by  omitted  explanatory  variables  within 
the  clustered  panel  data  and  a  population  averaged  logit  model  is  appropriate  for 
examining  across  clusters  of  panel  data.  Now  that  the  models  used  in  this  research  have 
been  justified,  it  is  necessary  to  look  at  the  results  that  the  various  logit  models  provide 
and  the  differences  between  them. 
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V.  INTERPRETATION  OF  EAC  SE/PM  COST  RATIO  ANALYSIS 


This  chapter  contains  a  detailed  analysis  of  the  data  and  provides  an  interpretation 
of  the  analysis  results.  This  first  section  contains  a  general  description  of  the  data 
collected.  The  next  three  sections  describe  the  relationship  between  major  defense 
acquisition  program  (MDAP)  cost  breaches  and  the  independent  variable  and  explanatory 
variables  that  incrementally  expanded  the  model.  These  three  sections  provide  equations 
that  model  the  variables  being  tested,  as  well  as  the  results  of  ordinary  least  squares 
(OLS)  and  logit  regression  analysis.  The  chapter  concludes  with  a  summary  of  the  most 
significant  data  analysis  results. 

A.  OBSERVED  MDAP  EAC  SE/PM  COST  RATIOS 

A  visual  representation  of  MDAP  cost  breaches  compared  to  MDAP  estimate  at 
completion  (EAC)  system  engineering/program  management  (SE/PM)  cost  ratios  is 
shown  in  Figure  5.  As  a  reminder,  in  this  report  cost  threshold  breaches  are  cost  overruns 
which  are  greater  than  or  equal  to  the  default  APB  cost  breach  threshold  value  of  10%. 
Therefore,  for  this  model  cost  breaches  include  APB  cost  breaches  and  Nunn-McCurdy 
cost  breaches,  since  the  Nunn-McCurdy  cost  breach  thresholds  are  all  greater  than  10%. 
More  specifically,  Figure  6  shows  the  total  number  of  cost  breaches  reported  by  the 
Selected  Acquisition  Reports  (SAR)  that  a  program  has  sustained  and  compares  it  against 
the  program’s  average  SE/PM  cost  ratio.  There  are  no  discernible  patterns  in  Figure  6,  so 
the  data  were  re-configured  to  see  whether  another  presentation  yielded  any  patterns  or  a 
different  interpretation. 


41 


Average  MDAP  EACSE/PM  Cost  Ratio 
versus  Cost  Breaches 
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Figure  5.  MDAP  EAC  SE/PM  Cost  Ratio  versus  Cost  Breaches 


Figure  5  is  based  on  the  cost  threshold  breach  and  corresponding  cost  data  that 
was  available  for  32  selected  MDAPs  since  1998.  From  this  representation  of  the  data,  it 
appears  that  more  than  half  of  the  MDAPs  maintain  an  EAC  SE/PM  cost  ratio  of  0.20  or 
less  and  have  experienced  less  than  two  cost  breaches  since  1998.  Furthermore,  it  can  be 
inferred  that  most  of  the  programs  have  experienced  at  least  one  cost  breach  and  these 
inferences  seems  to  confirm  a  recent  RAND  report  which  found  that  most  MDAPs  actual 
costs  exceeded  baseline  cost  estimates  (Arena  et  al.  2006). 14 

Observations  of  the  MDAP  SE/PM  cost  ratios  in  this  report  seem  to  concur  with 
the  RAND  study  which  suggests  that  trends  in  SE/PM  costs  vary  across  MDAPs  (Stem, 
Boito  and  Younossi  2006).  For  this  reason,  it  is  challenging  to  identify  relationships 
between  SE/PM  costs  and  cost  breaches.  The  average  EAC  SE/PM  cost  ratio  for  the 
sample  data  set  of  32  MDAPs  is  0.16.  This  suggests  that  on  average,  program  managers 
of  the  MDAPs  studied  in  this  report  spent  16%  of  their  budgets  on  systems  engineering 
and  program  management  activities.  Recall  that  the  systems  engineering  and  program 
management  costs  are  used  as  proxy  measures  of  the  transaction  costs  required  to 
administer  and  to  manage  the  MDAP. 

I4  For  most  of  the  programs  reviewed,  actual  costs  exceeded  the  baseline  cost  estimate  established  at 
Milestone  II  (program  initiation),  as  measured  by  the  cost  growth  factor  (Arena  et  al.  2006). 
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A  histogram  of  the  EAC  SE/PM  cost  ratios  observed  in  this  study  is  shown  in 
Figure  6.  From  this  histogram  we  can  observe  that  while  most  observed  ratios  varied 
between  0.05  and  0.35,  there  do  appear  to  be  two  outliers  having  values  of  0.50. 


EAC  SE/PM  Cost  Ratio  Histogram 
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Figure  6.  EAC  SE/PM  Cost  Ratio  Histogram  for  MDAPs 

We  assume  for  parametric  analysis  that  the  variables  are  normally  distributed 
because  one  of  the  underlying  assumptions  of  OLS  regression  is  that  the  data  is  normally 
distributed.  A  normal  probability  plot  (Figure  7)  was  used  to  evaluate  the  normality  of  the 
data.  This  plot  suggests  that  the  data  is  normally  distributed  with  an  r-squared  value  of 
0.85.'  While  this  normality  will  support  the  use  of  OLS  regression  in  our  analysis,  it  is 
not  required  for  logistic  regression. 


The  r-squared  value  measures  the  goodness  of  fit  of  the  data  to  the  trend  line  shown  in  Figure  7.  In 
a  perfect  dataset,  the  normal  probability  plot  of  normally  distributed  data  will  be  a  perfectly  straight  line 
with  an  r-squared  statistic  of  1.0.  The  dataset  used  in  this  report  is  confirmed  to  be  normally  distributed 
due  to  its  relatively  linear  normal  probability  plot  and  the  r-squared  statistic  of  the  trend  line  being  0.85. 
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Normal  Probability  Plot 


Figure  7.  Normal  Probability  Plot  of  EAC  SE/PM  Cost  Ratio 

In  Figure  8,  we  see  MDAP  EAC  SE/PM  cost  ratios  binned  by  various  numbers  of 
cost  threshold  breaches  for  selected  MDAPs  since  1998.  It  appears  that  the  information  in 
Figure  8  can  plausibly  be  described  by  a  simple  linear  regression  and  this  was  a  primary 
reason  that  the  linear  bivariate  and  multiple  regression  models  were  suggested  to  model 
the  data,  as  explained  in  the  previous  chapter.  If  this  is  a  valid  correlation,  it  supports  the 
hypothesis  that  “programs  with  higher  SE/PM  cost  ratios  will  experience  more  cost 
overruns  than  programs  with  lower  SE/PM  cost  ratios.”  Armed  with  this  information,  it 
is  logical  to  suspect  that  the  likelihood  of  a  program  sustaining  a  cost  threshold  breach 
could  be  associated  with  a  certain  SE/PM  cost  ratio.  The  null  hypothesis  is  that  the 
probability  of  a  cost  threshold  breaches  is  not  associated  with  the  MDAP  EAC  SE/PM 
cost  ratio  and  that  the  model  has  no  explanatory  power. 
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Trends  in  MDAP  EAC  SE/PM  Cost  Ratio 


Number  of  Cost  Breaches  for  MDAP 

■  EAC  SE/PM  Cost  Rato 


Figure  8.  Trends  in  MDAP  SE/PM  Cost  Ratios 

B.  COMPARISON  OF  MDAP  COST  BREACHES  TO  SE/PM  COST  RATIOS 

The  OLS  and  logistic  regression  techniques  that  were  discussed  in  the  previous 
chapter  will  be  utilized  and  some  of  the  strengths  and  weaknesses  of  these  models  will  be 
described.  The  most  valuable  information  obtained  from  these  models  is  the  marginal 
effects  and  average  marginal  effects  values.  The  marginal  effects  values  describe  the 
impact  of  the  independent  variable  on  the  dependent  variable.  For  example,  if  our 
analysis  were  to  conclude  that  the  average  marginal  effect  of  EAC  SE/PM  cost  ratio  was 
0.95,  then  the  model  is  stating  that  a  1.0%  change  in  the  independent  variable  (EAC 
SE/PM  cost  ratio)  will  result  in  a  0.95%  change  in  the  dependent  variable  (cost  breaches). 
It  is  important  to  understand  that  although  correlation  may  exist  between  the  dependent 
and  independent  variable,  causation  between  the  two  variables  cannot  be  implied. 

We  will  now  look  at  results  yielded  by  the  analysis  methods  that  were  described 
in  the  previous  chapter:  pooled  OLS  regression,  logit  fixed  effects  regression  and  logit 
population  averaged  regression.  In  the  first  series  of  model  runs,  the  variable  being  tested 
for  correlation  with  cost  breaches  is  the  EAC  SE/PM  cost  ratio.  For  the  remaining 
sections  of  this  chapter,  it  will  be  helpful  to  consider  the  following  linear  model  to 
explore  many  variables  and  cost  breaches: 
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Y  =  a  +  poX0  +  p,X, 


Variations  of  this  linear  model  will  also  be  helpful  for  interpreting  the  statistical 
tables  which  are  presented  in  this  section.  For  this  linear  model,  po  is  the  regression 
coefficient  that  will  be  associated  with  the  independent  variable  (EAC  SE/PM  cost  ratio) 
and  Pi  is  the  regression  coefficient  that  will  be  associated  with  all  other  explanatory 
variables.  In  general,  the  regression  coefficient  (PO  represents  the  amount  that  the 
dependent  variable  (Y)  and  explanatory  variable  (X;)  can  be  associated  to  a  unit  change 
in  the  independent  variable  EAC  SE/PM  cost  ratio  (Xo).  The  dependent  variable,  Y, 
represents  the  likelihood  of  a  cost  breach  occurring.  The  constant  (a)  represents  the  value 
of  the  probability  of  a  cost  breach  occurring  when  there  are  no  explanatory  variables  (X;) 
and  EAC  SE/PM  cost  ratio  (Xo)  is  zero  (Hutcheson  n.d.).  Accordingly,  for  the  first  series 
of  data  tests,  the  equation  for  the  model  being  tested  is: 

Y  =  a  +  poX0 

Xo  =  EAC  SE/PM  cost  ratio 

The  hypothesis  being  tested  is  that  the  probability  of  a  program  sustaining  a  cost 
threshold  breach  is  related  to  the  EAC  SE/PM  cost  ratio  of  the  MDAP.  The  null 
hypothesis  is  that  there  is  no  relation  between  the  probability  of  a  cost  threshold  breach 
and  the  EAC  SE/PM  cost  ratio  and  that  the  model  has  no  explanatory  power. 

The  results  of  all  three  of  the  regression  models  without  considering  any  other 
explanatory  variables,  such  as  the  time  since  program  initiation  (Milestone  B)  or  the  type 
of  contracts  within  the  program  are  shown  in  Table  4.  The  OLS  regression  and  logit 
population  averaged  regression  models  indicate  a  positive  relationship  that  is  significant 
at  the  5%  level  (p  =  0.05)  between  EAC  SE/PM  cost  ratio  and  the  likelihood  of  a  cost 
breach  occurring.  Since  we  established  in  the  previous  chapter  that  the  OLS  regression 
does  not  adequately  model  the  data  and  the  logit  fixed  effects  (FE)  model  is  not 
significant,  only  the  logit  population  averaged  (PA)  regression  results  will  be  further 
examined. 

The  marginal  effect  of  the  logit  population  averaged  model  shows  that,  based  on 
across  programs  observations,  for  every  unit  change  in  EAC  SE/PM,  the  log  odds  rise  in 
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the  probability  of  a  cost  breach  occurring  is  0.80. 16  This  means  that  for  every  1% 
increase  or  decrease  in  the  EAC  SE/PM  cost  ratio,  there  is  an  average  increase  or 
decrease  of  0.8%  in  the  probability  of  a  program  sustaining  a  cost  threshold  breach.  In 
logistic  regression,  the  value  of  the  marginal  effect  is  not  constant  and  must  be 
interpreted  accordingly.  For  example,  per  Table  4  when  the  EAC  SE/PM  cost  ratio  is  0.1 
the  marginal  effect  on  the  likelihood  of  a  cost  threshold  breach  is  0.76.  When  the  EAC 
SE/PM  cost  ratio  is  equal  to  0.1,  a  1%  increase  in  the  EAC  SE/PM  cost  ratio  will 
positively  correspond  to  a  0.76%  increase  in  the  likelihood  of  a  cost  threshold  breach 
occurring.  The  remaining  results  of  the  model  are: 

•  When  the  EAC  SE/PM  cost  ratio  is  0.2,  a  1%  change  in  EAC  SE/PM  cost 
ratio  positively  corresponds  to  a  0.90%  change  in  the  probability  of  a  cost 
breach  occurring. 

•  When  the  EAC  SE/PM  cost  ratio  is  0.3  a  1%  change  in  EAC  SE/PM  cost 
ratio  positively  corresponds  to  a  0.98%  change  in  the  probability  of  a  cost 
breach  occurring. 

•  When  the  EAC  SE/PM  cost  ratio  is  0.4  a  1%  change  in  EAC  SE/PM  cost 
ratio  positively  corresponds  to  a  1 .00%  change  in  the  probability  of  a  cost 
breach  occurring. 


16  The  logit  value  is  the  “log  odds”  value  and  is  mathematically  defined  to  be  the  natural  logarithm  of 
the  odds  ratio  that  a  certain  event  will  occur;  and  the  odds  ratio  is  simply  the  ratio  of  the  event  occurring 
versus  the  ratio  that  the  event  will  not  occur.  Logit  modeling  is  discussed  in  Chapter  IV,  Section  C 
“Approach  to  Analysis.” 
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EQUATION 

VARIABLES 

OLS 

Breach 

(Xo) 

Logit-FE 

Breach 

(X„) 

Logit-PA 

Breach 

(X„) 

EAC  SE/PM 

Cost  Ratio  (p0) 

1.097** 

(0.421) 

-9.238 

(7.825) 

4.006** 

(2.021) 

Constant  (a) 

0.143* 

(0.0789) 

-1.473*** 

(0.454) 

Number  of  Observations 

84 

39 

84 

R-squared  Statistic 

0.079 

Number  of  Programs 

32 

11 

32 

Robust  standard  errors  in  parentheses 

***  p<0.01,  ** 

p<0.05,  *  p<0.1 

Average  Marginal  Effect  in  PA  model  is  .8 

ME  @  EAC=0.1  0.76 
ME  @  EAC=0.2  0.90 
ME  @  EAC=0.3  0.98 

_ ME  g  EAC=0.4  1,00 _ 

Table  4.  Impact  of  EAC  SE/PM  with  No  Explanatory  Variables 

In  Table  4,  the  number  of  observations  represents  the  number  of  data  points  that 
were  tested  and  each  data  point  has  a  distinct  program  name  and  year.  The  “number  of 
programs”  is  the  number  of  programs  tested  in  each  model.  For  the  OLS  model,  all  of  the 
programs  were  tested.  The  r-squared  statistic  is  a  “goodness  of  fit”  value  for  the  model  to 
the  data  points  (no  such  statistic  exists  for  logistic  regression).  The  r-squared  value  is 
interpreted  to  mean  that  the  independent  variable,  EAC  SE/PM  cost  ratio,  explains  7.9% 
of  the  variance  in  this  model.  It  is  not  surprising  that  the  r-squared  statistic  is  so  small  and 
this  is  a  testament  that  the  single  variable  identified  in  the  model  (i.e.,  EAC  SE/PM  cost 
ratio)  does  not  explain  most  of  the  variation  in  the  dependent  variable.  The  robust 
standard  error,  which  is  the  square  root  of  the  estimate  of  variance  for  this  model,  is  used 
instead  of  the  standard  error  because  it  is  expected  to  be  more  accurate  when 
heteroskedasticity  and  autocorrelation  exist.  Heteroskedasticity  describes  the  variance  in 
the  error  terms  of  the  dependent  variable  as  the  value  of  the  explanatory  variable  changes 
(Barreto  and  Flowland  2006,  508-509).  Autocorrelation  is  the  correlation  of  a  variable 
value  with  itself  across  a  time  series. 

It  is  interesting  to  note  that  the  logit  fixed  effects  model  which  looks  within 

programs  does  not  show  correlation  between  EAC  SE/PM  and  cost  breaches.  This  seems 
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to  suggest  that  within  a  program  the  likelihood  of  a  program  incurring  a  cost  threshold 
breach  cannot  be  associated  to  the  EAC  SE/PM  cost  ratio.  This  may  be  because  the  fixed 
effect  model  uses  a  short  term  within  program  analysis  from  which  the  change  in 
likelihood  for  a  cost  breach  occurring  cannot  be  observed.  It  is  also  possible  that  no 
correlation  is  noted  in  the  fixed  effects  model  because  more  explanatory  variables  are 
needed  to  explain  the  relationship.  Nonetheless,  we  can  state  that  there  is  no  within- 
program  evidence  that  a  change  in  EAC  SE/PM  is  associated  with  a  change  in  the 
likelihood  of  a  cost  breach  occurring. 

The  logit  population  averaged  model  provides  a  long  term  look  across  programs 
and  yields  statistically  significant  results  for  associating  cost  breaches  to  EAC  SE/PM 
cost  ratio.  The  across  programs  look  can  be  interpreted  as  indiscriminately  considering  all 
Department  of  Defense  (DoD)  MDAPs  in  the  data  set,  regardless  of  the  program’s 
affiliation  with  a  specific  service  or  weapons  type.  The  results  suggest  that,  in  general, 
the  probability  of  a  cost  breach  occurring  is  related  to  the  EAC  SE/PM  cost  ratio.  That 
the  logit  population  averaged  model  and  the  OLS  model  both  show  a  statistically 
significant  relationship  between  EAC  SE/PM  and  cost  breaches  is  interesting;  however, 
as  discussed  in  the  previous  chapter,  the  OLS  regression  does  not  account  for  clustered 
data.  That  the  OLS  regression  and  logit  population  averaged  model  are  both  statistically 
significant  may  further  support  the  claim  that  when  considering  all  MDAPs,  any 
randomly  selected  program  is  likely  to  have  correlation  between  EAC  SE/PM  cost  ratio 
and  the  probability  of  a  cost  breach. 

According  to  Table  4,  the  marginal  effect  of  EAC  SE/PM  cost  ratio,  as  defined  in 
Chapter  III,  is  a  value  of  0.80  for  the  population  averaged  model,  for  an  almost  one  to  one 
correlation  between  changes  in  EAC  SE/PM  cost  ratio  and  changes  in  the  likelihood  of  a 
cost  breach.  Figure  9  illustrates  the  relationship  between  EAC  SE/PM  cost  ratio  and  cost 
breaches  for  the  population  averaged  model.  The  low  p-value  for  the  logit  population 
averaged  model  (p  <0.05)  confirms  the  statistical  significance  of  these  results.  In  other 
words,  we  can  assume  that  there  is  a  less  than  5%  possibility  that  the  results  yielded  in 
this  study  occurred  by  chance  (Thisted  2010).  Nonetheless,  provided  the  low  r-squared 
statistic  (0.079)  we  are  missing  some  of  the  explanatory  variables  that  contribute  to  the 
probability  a  cost  breach  occurring. 
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Figure  9.  Cost  Breach  versus  SE/PM  Logistic  Regression  S-Curve 

Interpretation  of  the  regression  s-curve  provided  (Figure  9)  may  lead  the  reader  to 
correctly  conclude  that  an  increase  or  decrease  in  the  EAC  SE/PM  cost  ratio  will  be 
associated  with  a  corresponding  increase  or  decrease  in  the  probability  of  a  cost  breach 
occurring.  Again,  it  is  necessary  to  recall  that  while  there  may  be  an  association  between 
EAC  SE/PM  cost  ratio  and  cost  breaches,  there  is  no  evidence  which  suggest  or  supports 
any  claim  that  the  EAC  SE/PM  is  the  causal  factor  for  a  cost  threshold  breach. 

The  analysis  suggests  that  there  is  a  possible  relation  between  SE/PM  cost  ratio 
and  the  likelihood  of  occurrence  of  a  cost  threshold  breach.  For  example,  if  an  MDAP 
has  a  SE/PM  cost  ratio  of  0.20  we  expect  a  30%  likelihood  of  incurring  a  cost  breach.  If 
that  same  MDAP’s  SE/PM  cost  ratio  were  to  increase  to  0.40,  the  analysis  suggests  a 
corresponding  increase  in  the  probability  of  a  cost  breach  occurring,  to  a  60%  chance  of 
incurring  a  cost  breach.  Notwithstanding,  there  is  no  evidence  that  the  EAC  SE/PM  cost 
ratio  should  be  considered  a  causal  factor  for  cost  breaches,  merely  an  indicator. 

C.  COMPARISON  OF  MDAP  COST  BREACHES  TO  SE/PM  COST  RATIOS 

AND  PROGRAM  MATURITY 

The  logarithm  of  program  maturity,  considered  in  this  report  to  be  the  length  of 
time  (in  years)  since  the  requisite  program  contracts  entered  Milestone  B,  was  used  to 
expand  the  model.  As  discussed  in  Chapter  III,  this  definition  of  maturity  also 
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incorporates  attainment  of  the  DoD  requirements  for  an  MDAP  to  accomplish  Milestone 
B  approval,  namely  that  the  design  and  technology  utilized  in  the  project  are  mature.  The 
rationale  behind  measuring  the  effect  of  log  maturity  against  cost  breaches  is  that  we 
expect  older  programs  to  have  less  uncertainty  and  risk  than  younger  programs.  It  is 
presumed  that  older  programs  have  more  experienced  managers  and  more  streamlined 
and  efficient  processes  as  compared  to  a  younger  program,  thus  making  older  programs 
less  likely  to  sustain  a  cost  breach.  Taking  the  natural  log  of  maturity  helps  to  linearize 
the  relation  of  maturity  to  cost  breaches  in  this  model.  In  this  section,  we  will  look  at  the 
impact  of  EAC  SE/PM  cost  ratio  on  the  likelihood  of  a  cost  breach  occurring  while 
including  the  explanatory  variable  “Log  Maturity,”  the  logarithm  of  the  maturity  variable: 

Y  =  a  +  poX0  +  piXj 

X0  =  EAC  SE/PM  Cost  Ratio 

Xi  =  Log  Maturity 

The  null  hypothesis  is  that  there  is  no  relation  of  cost  breaches  with  EAC  SE/PM 
cost  ratio  and  program  maturity  as  defined  in  this  report.  The  analysis  is  modeled  by  logit 
fixed  effects  and  logit  population  averaged  models  and  the  resulting  impact  of  the 
explanatory  variable  Log  Maturity  is  displayed  in  Table  5. 


Logit-FE 

Logit-PA 

VARIABLES 

(X0 

(X0 

Log  Maturity  (Pi) 

0.429 

0.350 

(0.877) 

(0.302) 

EAC  SE  PM 

-8.826 

4.449** 

Cost  Ratio  (Po) 

(7.897) 

(1.907) 

Constant  (a) 

-2.193*** 

(0.752) 

Number  of  Observations 

39 

84 

Number  of  Programs 

11 

32 

Standard  errors  in  parentheses 

***p<0.01,  ** 

p<0.05.  *  p<0.1 

Average  marginal  effect  in  PA  of  Log  maturity  is  0.7 

and  EAC  SE  PM  Cost  Ratio  is  0.89 

ME  @  EAC=0.1  0.83 

ME  @  EAC=0.2  0.99 

ME  @  EAC=0.3  1.08 

ME  @  EAC=0.4  1.08 

Table  5.  Impact  of  EAC  SE/PM  with  Log  Maturity 
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While  the  logarithm  of  the  length  of  the  program  does  not  have  a  statistically 
significant  effect  in  all  of  the  models,  the  average  marginal  effect  of  Log  Maturity  in  the 
population  averaged  logit  model  is  0.70  and  for  EAC  SE/PM  cost  ratio  the  average 
marginal  effect  is  0.89.  This  means  that  a  one  unit  (1%)  change  in  the  Log  Maturity  of  an 
MDAP  equates  to  a  0.70%  change  in  the  likelihood  of  a  cost  breach  occurring.  Similarly, 
a  one  unit  (1%)  change  in  the  EAC  SE/PM  cost  ratio  will  result  in  a  0.89%  change  in  the 
likelihood  of  an  MDAP  incurring  a  cost  breach.  The  remaining  results  of  the  model  are: 

•  When  the  EAC  SE/PM  cost  ratio  is  0.1,  a  1%  change  in  EAC  SE/PM  cost 
ratio  positively  corresponds  to  a  0.83%  change  in  the  probability  of  a  cost 
breach  occurring. 

•  When  the  EAC  SE/PM  cost  ratio  is  0.2,  a  1%  change  in  EAC  SE/PM  cost 
ratio  positively  corresponds  to  a  0.99%  change  in  the  probability  of  a  cost 
breach  occurring. 

•  When  the  EAC  SE/PM  cost  ratio  is  0.3,  a  1%  change  in  EAC  SE/PM  cost 
ratio  positively  corresponds  to  a  1 .08%  change  in  the  probability  of  a  cost 
breach  occurring. 

•  When  the  EAC  SE/PM  cost  ratio  is  0.4,  a  1%  change  in  EAC  SE/PM  cost 
ratio  positively  corresponds  to  a  1 .08%  change  in  the  probability  of  a  cost 
breach  occurring. 

The  practical  interpretation  of  these  results  may  be  that  SE/PM  costs  become  a 
larger  percentage  of  the  total  program  costs  due  to  the  proactive  or  reactive  actions  of  the 
program  manager  to  avoid  a  cost  threshold  breach.  Thus,  there  is  evidence  to  suggest  that 
the  SE/PM  cost  ratio  may  be  an  indicator  of  cost  troubles. 

D.  COMPARISON  OF  MDAP  COST  BREACHES  TO  SE/PM  COST  RATIOS, 

PROGRAM  MATURITY  AND  CONTRACT  TYPE 

We  will  now  examine  the  impact  of  the  type  of  contract  on  the  likelihood  of  a  cost 
breach  occurring.  Contract  type  was  included  because  it  is  generally  accepted  by 
acquisitions  professionals  that  firm  fixed  price  contracts  are  frequently  negotiated 
between  the  government  and  contractors  for  less  risky  programs  and  cost-plus  contracts 
are  negotiated  for  projects  which  have  greater  uncertainty  and  are  perceived  to  be  more 
risky  (Government  Accountability  Office  2009b).  Much  of  the  rationale  for  associating 
contract  types  and  projects  based  on  the  perceived  riskiness  of  the  projects  is  rooted  in 
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the  cost  sharing  and  incentives  offered  by  the  different  contract  types.17  Nonetheless,  it 
seems  logical  that  riskier  MDAPs  should  have  cost-plus  contracts  and  experience  more 
cost  breaches  than  their  less  risky  counterparts.  The  null  hypothesis  of  this  analysis  is  that 
there  is  no  relation  between  cost  breaches  and  EAC  SE/PM  cost  ratio,  log  maturity  and 
MDAP  contract  type.  The  linear  equation  model  which  analyzes  the  impact  of  EAC 
SE/PM  on  the  likelihood  of  a  cost  breach  occurring  while  including  the  explanatory 
variables  Log  Maturity  and  MDAP  contract  type  is: 

Y  =  a  +  poX0  +  piX1  +  p2X2 

X0  =  EAC  SE/PM  Cost  Ratio 

Xj  =  Log  Maturity 

X2  =  Contract  Type 

For  this  model,  firm- fixed  price  contracts  have  p2=0  and  cost-plus  contracts  have  (32=1. 


17  In  a  firm  fixed-price  contract  there  is  not  cost  sharing  between  the  government  and  the  contractor 
and  the  contractor  has  full  responsibility  for  the  performance  costs  and  resulting  profit  (or  loss).  In  a  cost- 
plus  contract,  a  share  ratio  based  on  the  contract  cost  and  the  contractor’s  fee  (profit)  is  negotiated  so  that 
the  contractor  has  a  pre-determined  responsibility  for  the  performance  costs  which  will  directly  affect  the 
fee  (profit).  In  the  case  of  both  contract  types,  incentives  may  be  offered  in  which  the  contractor’s 
responsibility  for  the  performance  costs  and  the  profit  or  fee  incentives  offered  are  tailored  to  the 
uncertainties  involved  in  contract  performance  (General  Services  Administration  2005). 
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Logit  PA  (Xi) 
VARIABLES  “Fixed  Price" 

(P:=0) 

Logit  PA  (Xi) 
“Cost  Plus" 

(P:=l) 

Log  Maturin'  (Pi) 

-0.0967 

1.115** 

(0.586) 

(0.553) 

EAC  SE  PM  (Po) 

-2.121 

5.172* 

Cost  Ratio 

(4.665) 

(2.846) 

Constant  (a) 

-0.906 

-3.255*** 

(1-346) 

(1-214) 

Number  of  Observations 

37 

41 

Number  of  Programs 

15 

16 

Robust  standard  errors  in  parentheses 


***  p<0.01,  **  p<0.05,  *  p<0.1 

Including  ME  for  the  type  1  model  only 
Average  marginal  effect  in  PA  of  Log  maturity  is  .21.  and  EACSEP  is  1.00 

ME  g  EAC=0.1  0.94 
ME  g  EAC=0.2  1.08 
ME  @EAC=0.3  1.12 

_ ME  g  EAC=0.4  1.07 _ 


Table  6.  Logistic  Regression  with  Log  Maturity  and  Contract  Type 


Noting  the  results  summarized  in  Table  6  and  considering  the  independent  and 
explanatory  variables  in  this  model,  the  statistically  significant  results  are  observed  in 
programs  with  cost-plus  contracts.  This  observation  is  statistically  significant  at  the  10% 
level.  Since  only  cost-plus  programs  yield  statistically  significant  results,  the  marginal 
effects  of  the  independent  and  explanatory  variables  have  only  been  measured  for 
programs  with  cost-plus  contracts.  That  the  model  does  not  produce  statistically 
significant  results  for  programs  with  fixed-price  contracts  implies  the  existence  of 
additional  explanatory  variables  which  are  not  captured  within  the  scope  of  this  report. 
For  the  cost-plus  contract  programs,  according  to  the  population  averaged  model,  the 
average  marginal  effect  of  Log  Maturity  is  0.21  and  for  EAC  SE/PM  cost  ratio  is  1.00. 
This  means  that  for  a  one  unit  (1%)  change  in  the  Log  Maturity  the  likelihood  of  a  cost 
breach  occurring  will  change  by  0.21%.  Moreover,  for  a  one  unit  (1%)  change  in  the 
EAC  SE/PM  cost  ratio  the  likelihood  of  a  cost  breach  occurring  will  change  by  1.00%. 
Similar  to  the  results  in  Section  C  of  this  chapter,  as  the  EAC  SE/PM  cost  ratio  changes 
from  0.1  to  0.4  there  are  corresponding  changes  in  the  marginal  effect  on  the  probability 
of  a  cost  breach  occurring  in  a  program: 
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•  When  the  EAC  SE/PM  cost  ratio  is  0.1,  a  1%  change  in  the  EAC  SE/PM 
cost  ratio  positively  corresponds  to  a  0.94%  change  in  the  probability  of  a 
cost  breach  occurring. 

•  When  the  EAC  SE/PM  cost  ratio  is  0.2,  a  1%  change  in  the  EAC  SE/PM 
cost  ratio  positively  corresponds  to  a  1 .08%  change  in  the  probability  of  a 
cost  breach  occurring. 

•  When  the  EAC  SE/PM  cost  ratio  is  0.3,  a  1%  change  in  the  EAC  SE/PM 
cost  ratio  positively  corresponds  to  a  1.12%  change  in  the  probability  of  a 
cost  breach  occurring. 

•  When  the  EAC  SE/PM  cost  ratio  is  0.4,  a  1%  change  in  the  EAC  SE/PM 
cost  ratio  positively  corresponds  to  a  1 .07%  change  in  the  probability  of  a 
cost  breach  occurring. 

E.  SUMMARY  OF  RESULTS 

In  this  chapter  we  looked  at  the  SE/PM  ratios  and  cost  breaches  for  32  MDAPs 
during  the  period  of  1998  to  2011.  We  noted  that  the  observed  SE/PM  cost  ratios  were 
approximately  normally  distributed  with  an  average  of  0.16  and  a  range  of  0.05  to  0.5. 
Our  graphical  analysis  indicated  that  there  appears  to  be  a  relationship  between  SE/PM 
cost  ratios  and  cost  breaches  as  shown  in  Figure  8.  We  used  multiple  models  and  three 
types  of  regression  to  further  examine  this  relationship:  ordinary  least  squares  (OLS) 
regression,  fixed  effects  (FE)  logit  and  population  averaged  (PA)  logit,  as  described  in 
Chapter  IV.  A  linear  model  was  used  to  describe  the  relationship: 

Y  =  a  +  p0X0  +  piX1  +  p2X2 

X0  =  EAC  SE/PM  Cost  Ratio 

Xi  =  Log  Maturity 

X2  =  Contract  Type 

Based  on  the  results  of  the  analysis,  using  the  population  averaged  logit  model  to 
look  across  programs  seems  most  useful.  This  is  probably  due  to  the  population  averaged 
model  making  longer  term  observations,  whereas  the  fixed  effects  model  is  making  short 
tenn  observations.  Of  the  three  variables  tested  using  the  various  logistic  regression 
techniques,  the  model  which  included  the  explanatory  variables  Log  Maturity  and  cost- 
plus  contract  types  had  the  largest  values  for  the  average  marginal  effects  of  EAC  SE/PM 
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cost  ratio:  the  average  marginal  effect  of  Log  Maturity  is  0.21  and  for  EAC  SE/PM  cost 
ratio  is  1.00.  Furthennore,  in  this  model  as  the  EAC  SE/PM  cost  ratio  changes  from  0.1 
to  0.4  there  are  corresponding  changes  in  the  marginal  effect  on  the  probability  of  a  cost 
breach  occurring  in  a  program. 

The  analysis  results  seem  to  support  the  hypothesis  that  high  transaction  costs  are 
an  ex-ante  indicator  related  to  MDAP  cost  breaches.  The  next  chapter  will  discuss 
conclusions  that  can  be  drawn  from  these  results,  as  well  as  research  areas  which  were 
not  captured  in  this  report  that  may  contribute  to  improving  DoD  cost  estimation. 
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VI.  SUMMARY  AND  FUTURE  RESEARCH 


This  chapter  of  the  report  provides  an  overview  of  the  salient  observations  of  this 
research.  This  includes  inferences  that  can  be  made  based  on  the  key  takeaways  from  the 
logit  population  averaged  regression,  which  turned  out  to  be  the  best  method  of  approach 
that  was  considered  for  modeling  the  data.  As  this  research  only  addresses  the  role  that 
transaction  costs  play  in  Department  of  Defense  (DoD)  cost  estimates,  this  chapter  also 
presents  areas  for  further  research  that  may  help  system  engineers  and  cost  analysts  to 
improve  program  cost  estimation  and  possibly  use  breaches  to  create  a  predictive  model 
in  the  future. 

A.  CONCLUSIONS 

This  analysis  is  based  on  the  assumption  that  there  is  a  real  cost  for  business 
negotiations  and  managing  the  program,  commonly  known  as  transaction  costs.  We  seek 
to  determine  (1)  if  there  is  a  correlation  between  these  transaction  costs  and  acquisition 
cost  threshold  breaches;  and  (2)  if  cost  threshold  breaches  are  related  to  the  amount,  or 
proportion  of  total  costs,  that  are  spent  on  program  transaction  costs.  The  data  set 
included  major  defense  acquisition  programs  (MDAPs)  with  10%  (or  greater)  cost 
threshold  breaches.  Using  logit  population  averaged  regression  analysis  yielded 
statistically  significant  results  for  the  programs  with  cost-plus  contracts.  These  contracts 
exhibited  relationships  between  (1)  the  estimate  at  completion  (EAC)  system 
engineering/program  management  (SE/PM)  cost  ratio  and  the  likelihood  of  a  cost  breach 
occurring;  and  (2)  the  log  maturity  and  the  likelihood  of  a  cost  breach  occurring. 

The  results  show  that  as  the  EAC  SE/PM  cost  ratio  rises  between  0.1  and  0.3 
there  is  a  statistically  significant  corresponding  increase  in  the  probability  of  a  cost 
threshold  breach  occurring.  This  provides  answers  to  both  of  these  research  questions. 
First,  the  statistically  significant  correspondence  between  the  variables  infers  that 
correlation  exists.  Second,  the  relation  between  cost  threshold  breaches  and  the  amount  of 
transaction  costs  is  confirmed  because  as  the  relative  amount  of  transaction  costs  for  an 
MDAP  increases,  the  likelihood  of  the  occurrence  of  a  cost  threshold  breach  also 
increases. 
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The  report  also  sought  to  determine  whether  underestimated  transaction  costs  had 
contributed  to  MDAP  cost  threshold  breaches.  While  the  research  established  correlation 
between  the  independent  and  dependent  variables  there  was  no  testing  for  causality.  The 
existence  of  a  correlation  between  transaction  costs  and  MDAP  cost  threshold  breaches 
neither  suggests  that  increased  transaction  costs  caused  the  cost  threshold  breach,  nor  that 
a  cost  threshold  breach  generated  increased  transaction  costs.  Without  establishing 
causality,  it  cannot  conclusively  be  stated  that  underestimated  transaction  costs  have 
contributed  to  cost  growth  for  MDAPs. 

B.  IMPLICATIONS  TO  SYSTEMS  ENGINEERING 

It  is  incumbent  upon  system  engineers  (SEs)  to  understand  the  cost  estimation 
process  for  any  project  or  program  in  order  to  provide  support  to  the  program  manager. 
The  SE  process  is  the  overarching  framework  designed  to  be  the  “integrating  mechanism 
for...  limitations  imposed  by  technology,  budget,  and  schedule”  (Defense  Acquisition 
University  2011b,  B-264).  For  the  SE  effort  to  be  effective,  understanding  the  impact  of 
transaction  costs  on  the  acquisition  cost  of  systems  is  important.  Our  research  informs  the 
SE  process  by  suggesting  an  association  between  transaction  costs  and  cost  threshold 
breaches  for  major  defense  acquisition  programs  (MDAPs)  with  cost-plus  contracts. 

C.  AREAS  FOR  FURTHER  RESEARCH 

The  logit  population  averaged  model  proved  to  be  useful  in  this  research  for 
multiple  reasons.  Two  of  the  more  salient  reasons  are  that  (1)  logistic  regression  models 
are  designed  for  binary-outcome  dependent  variables;  and  (2)  the  data  is  in  panel  form,  as 
discussed  in  Chapter  IV.  This  research  strongly  suggests  that  of  the  three  models 
considered,  the  logit  population  averaged  model  would  be  useful  for  future  research — 
particularly  for  estimating  the  marginal  effect  of  the  independent  variable(s)  on  the 
dependent  variable.  That  is,  the  logit  population  averaged  model  would  be  useful  for 
testing  the  impact  of  EAC  SE/PM  cost  ratio  with  other  explanatory  variables  against  the 
probability  of  a  cost  threshold  breach  occurring. 

It  would  be  interesting  to  run  logit  regressions  with  more  variables  in  order  to 

increase  the  explanatory  power  of  the  model  and  determine  other  factors  which  may 
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contribute  to  cost  threshold  breaches.  That  the  “maturity”  of  the  program,  as  defined  in 
Chapter  IV  (Section  B)  of  this  report,  is  related  to  the  occurrence  of  a  cost  threshold 
breach  is  interesting  and  may  be  worthy  of  further  research.  In  particular,  the  correlation 
of  log  maturity  paired  with  other  variables  to  cost  breaches  has  not  yet  been  explored. 
Whether  by  using  actual  data  or  proxies,  future  research  of  the  following  for  correlation 
to  cost  threshold  breaches  may  prove  interesting:  (1)  program  complexity;  (2)  “to  date” 
SE/PM  cost  ratio;  and  (3)  the  degree  to  which  programs  are  ahead  of  or  behind  on 
schedule. 

Tracking  MDAP  SE/PM  cost  ratio  trends  over  time  to  better  understand  the 
implications  of  pre-breach  SE/PM  cost  ratios  and  post-breach  SE/PM  cost  ratios  may  also 
prove  useful  to  program  managers.  Perhaps  the  pre-breach  SE/PM  cost  ratio  when  compared 
against  program  acquisition  milestones  through  time  is  indicative  of  future  breaches.  Is  it 
possible  that  the  SE/PM  cost  ratio  is  a  resource  tail  for  mitigating  actions  that  a  program 
manager  may  take  to  avoid  or  lessen  the  severity  of  a  cost  threshold  breach? 

Ultimately,  considering  the  association  of  these  independent  variables  with  the 
dependent  variable  will  be  integral  to  developing  a  multivariate  model  that  has  predictive 
power  for  MDAP  cost  threshold  breaches.  Such  a  predictive  model  would  be  an 
invaluable  tool  for  MDAP  stakeholders  and  particularly  systems  engineers  and  program 
managers.  It  is  likely  that  any  future  research  along  the  lines  essayed  in  this  study 
involves  revisiting  the  basic  model — which  is  depicted  in  Figure  2.  While  it  proved 
useful,  it  (like  all  models)  is  a  simplification  of  the  real  world.  And,  it  is  likely  this 
particular  effort  has  wrung  out  about  all  that  model  can  provide  by  way  of  insights  and 
explanatory  power.  Hence,  any  significant  advances  in  the  understanding  of  transaction 
costs  in  defense  costs,  and  their  estimation,  probably  entails  the  formulation  of  a  richer 
(and  likely  more  complex)  model.  That  said,  we  feel  that  Figure  2  does  provide  a  useful 
starting  point  for  devising  a  better  model. 
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APPENDIX 


A.  DEFINITION  OF  TERMS  AND  VARIABEES 

All  definitions  from  DAU  Glossary,  2011  found  at  (https://dap.dau.mil/glossary/ 
Pages/Default. aspx)  unless  otherwise  indicated. 


Acquisition  Cost  Estimate 


Component  Cost  Estimate 
(CCE). 


Equal  to  the  sum  of  the  development  cost  for  prime 
mission  equipment  and  support  items;  the 
procurement  cost  for  prime  mission  equipment, 
support  items,  and  initial  spares;  and  the  system 
specific  facilities  cost. 

The  generic  term  — DoD  Component  Cost  Estimate! 
is  used  to  provide  considerable  latitude  to  each 
military  service  or  defense  agency  as  to  the  actual 
responsibility  for  this  cost  estimate.  In  some  cases,  a 
military  service  assigns  the  responsibility  to  the 
Program  Office  (PO),  which  then  provides  a 
Program  Office  Life  Cycle  Cost  Estimate  (PLCCE). 
In  other  cases,  the  DoD  Component  may  adopt  a 
more  corporate  approach  in  which  an  initial  PO  cost 
estimate  is  subject  to  considerable  review  and 
possible  adjustment  as  determined  by  the  Service 
Cost  Center  or  defense  agency  equivalent  (Defense 
Acquisition  Guidebook  2011a). 


Cost  Estimate  A  judgment  or  opinion  regarding  the  cost  of  an 

object,  commodity,  or  service.  A  result  or  product  of 
an  estimating  procedure  that  specifies  the  expected 
dollar  cost  required  to  perfonn  a  stipulated  task  or  to 
acquire  an  item.  A  cost  estimate  may  constitute  a 
single  value  or  a  range  of  values. 

Estimate  at  Completion  Actual  direct  costs,  plus  indirect  costs  or  costs 

(EAC)  allocable  to  the  contract,  plus  the  estimate  of  costs 

(direct  and  indirect)  for  authorized  work  remaining. 


Independent  Cost  Estimate  A  Life  Cycle  Cost  Estimate  (LCCE)  for  Acquisition 

(ICE).  Category  (ACAT)  I  programs  prepared  by  an  office 

or  other  entity  not  under  the  supervision,  direction, 
or  control  of  the  military  department,  defense 
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agency,  or  other  Component  of  DoD  that  is  directly 
responsible  for  B-116  development  or  acquisition  of 
the  program,  or  if  the  decision  authority  has  been 
delegated  to  a  Component,  prepared  by  an  office  or 
other  entity  that  is  not  directly  responsible  for 
carrying  on  the  development  or  acquisition  of  the 
program. 


Independent  Government 
Cost  Estimate  (IGCE). 


Life  Cycle  Cost 


Nunn-McCurdy  Breach 


Operation  and  Maintenance 
(O&M) 


An  estimate  of  the  cost  for  goods  and/or  estimate  of 
services  to  be  procured  by  contract.  Such  estimates 
are  prepared  by  government  personnel,  i.e., 
independent  of  contractors. 

For  a  defense  acquisition  program,  LCC  consists  of 
Research  and  Development  (R&D)  costs,  investment 
costs,  operating  and  support  costs,  and  disposal  costs 
over  the  entire  life  cycle.  These  costs  include  not 
only  the  direct  costs  of  the  acquisition  program,  but 
also  indirect  costs  that  would  be  logically  attributed 
to  the  program.  In  this  way,  all  costs  that  are 
logically  attributed  to  the  program  are  included, 
regardless  of  funding  source  or  management  control 
(Defense  Acquisition  Guidebook  2011a). 

Refers  to  Title  10  U.S.C.  §  2433,  Unit  Cost  Reports 
(UCRs).  This  amendment  to  Title  10  was  introduced 
by  Senator  Sam  Nunn  and  Representative  Dave 
McCurdy  in  the  National  Defense  Authorization  Act 
(NDAA)  for  Fiscal  Year  (FY)  1982.  Requires  that 
Acquisition  Category  I  (ACAT  I)  Program  Managers 
(PMs)  maintain  current  estimates  of  Program 
Acquisition  Unit  B-174  Cost  (PAUC)  and  Average 
Procurement  Unit  Cost  (APUC).  If  the  PAUC  or 
APUC  increases  by  25  percent  or  more  over  the 
current  Acquisition  Program  Baseline  (APB) 
objective,  or  50  percent  or  more  over  the  original 
APB  objective,  the  program  must  be  terminated 
unless  the  Secretary  of  Defense  (SECDEF)  certifies 
to  Congress  that  the  program  is  essential  to  national 
security  (DoDI  5000.02  and  Defense  Acquisition 
Guidebook,  Chapter  10). 

O&M  appropriations  fund  expenses  such  as  civilian 
salaries,  travel,  minor  construction  projects, 
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Procurement 


Program  Office  Estimate 
(POE). 

Research,  Development, 
Test,  and  Evaluation 
(RDT&E) 


SE/PM  Cost 


SE/PM  Ratio 


operating  military  forces,  training  and  education, 
depot  maintenance,  stock  funds,  and  base  operations 
support. 


Procurement  appropriations  fund  those  acquisition 
programs  that  have  been  approved  for  production  (to 
include  Low-Rate  Initial  Production  (LRIP)  of 
acquisition  objective  quantities),  and  all  costs 
integral  and  necessary  to  deliver  a  useful  end  item 
intended  for  operational  use  or  inventory  upon 
delivery. 

A  Component  Cost  Estimate  (CCE)  of  Life  Cycle 
Costs  (LCCs)  conducted  by  an  acquisition  Program 
Office  (PO). 

RDT&E  appropriations  fund  the  efforts  performed 
by  contractors  and  government  activities  required  for 
the  Research  and  Development  (R&D)  of 
equipment,  material,  computer  application  software, 
and  their  Test  and  Evaluation  (T&E)  including 
Initial  Operational  Test  and  Evaluation  (IOT&E)  and 
Live  Fire  Test  and  Evaluation  (LFT&E).  RDT&E 
also  funds  the  operation  of  dedicated  R&D 
installation  activities  for  the  conduct  of  R&D 
programs. 

This  cost  is  the  sum  of  recurring  and  non-recurring 
costs  for  the  SE/PM  WBS  line  item.  This  cost  is  not 
inclusive  of  the  contractor’s  or  subcontractor’s 
profit/loss  or  fees. 

This  ratio  serves  as  a  proxy  for  transaction  costs  and 
is  defined  to  be  the  program’s  SE/PM  costs  divided 
by  total  cost: 


SE/PM  Ratio  = 


"To  Date"  SE/PM  Costs 
Total  Cost  EAC 
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B.  USING  THE  DEFENSE  ACQUISITION  MANAGEMENT  INFORMATION 

RETRIEVAL  SYSTEM 

Establishing  an  account  with  the  Defense  Acquisition  Management  Information 
Retrieval  (DAMIR)  system  was  a  fairly  simple  process.  Individual  institutions  are 
assigned  DAMIR  points  of  contact  (POCs)  who  are  responsible  for  requesting  a  DAMIR 
account  on  behalf  of  the  user.  The  user  is  responsible  for  providing  the  requisite  personal 
information  and  justification  for  use  to  the  institution’s  DAMIR  POC.  The  DAMIR  help 
desk  (phone  number  found  at  http://www.acq.osd.mil/damir/index.html)  was  helpful  for 
establishing  the  DAMIR  account. 

Generally,  the  organized  and  established  criteria  for  SAR  submissions  facilitated 
the  effort  for  finding  MDAPs  and  their  associated  SARs,  but  it  is  important  to  note  that 
political  events  can  also  affect  SAR  submission.  For  instance,  in  years  which  the 
president’s  budget  was  signed  late  (e.g.,  2000  and  2008)  the  only  SARs  found  are  for 
MDAPs  which  met  the  mandate  to  submit  an  interim  quarterly  SAR,  such  as  a  Nunn- 
McCurdy  breach  or  schedule  delay  of  at  least  6  months. 
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C.  USING  THE  DEFENSE  COST  AND  RESOURCE  CENTER 

It  requires  approximately  one  week  to  establish  a  Defense  Cost  and  Resource 
Center  (DCARC)  account  and  access  the  cost  and  software  data  reporting  (CSDR) 
database.  Using  the  online  registration  instructions  found  at  the  DCARC  Knowledge 
Portal  (http://dcarc.cape.osd.mil/)  made  the  process  simple.  The  access  levels  that  are 
required  to  view  the  CSDR  information  used  in  this  report  are  “DACIMS35_Analyst” 
and  “EVM_Analyst.”  Once  the  user’s  CAC  card  is  registered  with  DCARC,  access  will 
be  granted  to  the  database.  Access  to  DCARC  also  includes  access  to  the  Defense 
Automated  Cost  Information  Management  Systems  (DACIMS)  3.5  Library.  The  CSDR 
library  contains  folders  of  active  and  inactive  MDAP  contract  data  sorted  by  weapon 
system  types.  More  specifically,  contract  information  within  each  of  these  folders 
includes:  the  DD  Form  1921  (Cost  Summary  Data  Report),  the  DD  Form  1921-1 
(Functional  Cost-Hour  Report)  and  the  contract  work  breakdown  structure  (CWBS) 
Index  and  Dictionary  among  other  documents. 

1.  Sample  Cost  Data  Summary  Report  (DD  Form  1921) 

The  cost  data  summary  report  (Figure  10)  comprises  many  categories  of  program 
metadata  and  reported  data.  A  list  of  these  categories  and  detailed  information  about  the 
DD  Form  1921  can  be  found  in  DI-FNCL-81565C. 
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SECURITY  CLASSIFICATION. 


COST  DATA  SUMMARY  REPORT 


Form  Approved 
OMB  NO.  0704-0186 


The  public  reporting  burden  lor  Ibis  collection  ol  information  is  estmated  to  average  8  hours  per  response,  including  the  time  lor  reviewing  instructions,  searching  existing  data  sources,  gatiernq  and  mam  tan  ng  the  data  needed  and  compiebng  and  reviewing  the  colection  o  I  information  Send 
comments  regarding  this  burden  estimate  or  any  other  aspect  ot  this  collection  ot  information,  including  suggestions  lor  reducing  the  burden,  to  Department  ot  Defense.  Washington  Headquarters  Semces,  Executive  Services  Directorate  (0704  0188).  Respondents  should  be  aware  that 
notwithstanding  any  other  provision  ot  law.  no  poison  shall  be  subnet  to  any  penalty  lor  tailing  to  comply  with  a  collection  ot  information  if  It  does  nor  display  a  currently  valid  OMB  control  number 
PLEASE  DO  NOT  RETURN  YOUR  COMPLETED  FORM  TO  THE  ABOVE  ORGANIZATION. 


1.  PROGRAM 

2.  PRIME  MISSION  PRODUCT 

J.  CONTRACTOR  TYPE  (XofW 

4.  N  AM  E  A D  D R ESS  i Indude  ZIP  Code) 

5.  APPROVED  PLAN  NUMBER 

a.  MDAP: 

|  PRIME  ASSOCIATE 

b.  PHASE: 

|  DIRECT  REPORTING  SUBCONTRACTOR 

6.  CUSTOMER  f Orroci -flaps  mrij SuOcorxradcv trim  On/y) 

7.  CONTRACT  TYPE 

8.  CONTRACT  PRICE 

9.  CONTRACT  CEILING 

10.  TYPE  ACTION 

a.  CONTRACT  NO..  c.  SOLICITATION  NO.: 

b.  LATEST  MODIFICATION.  d.  NAME; 

11.  PERIOD  OF  PERFORMANCE 

a.  START  DATE  (YYYYMMDD): 

b.  END  DATE  (YYYYMMDD): 

VT  NAME  (Last.  Fust.  M.ddlo  In/Mlt 


12.  APPROPRIATION 

RDT4E 

_  PROCUREMENT 

OAM 

18.  DEPARTMENT 


13.  REPORT  CYCLE  IX one)  14.  SUBMISSION  NUMBER 

_  INITIAL 

_  INTERIM 

riNAL 

I  19.  TELEPHONE  NUMBER  (Indudo  Area  Code)  I  20.  EMAIL  ADDRESS 


IS.  RESUBMISSION  NUMBER 


16.  REPORT  AS  OF  ( YYYYMMDD ) 


[21.  DATE  PREPARED  (YYYYMMDO) 


WBS  1  1  NUMBER  OF  I  COSTS  INCURRED  TO  DATE 

I  NUMBER  OF  1 

COSTS  INCURRED  AT  COMPLETION 

CODE  TO  DATE  NONRECURRING 

A  B  C  0 

RECURRING 

E 

TATI1  UNITS  AT 

TOTAL  COMPLETION 

F  G 

NONRECURRING  RECURRING  TOTAL 

H  1  J 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

22.  REMARKS 

68 


2.  Steps  for  Estimate  at  Completion  SE/PM  Cost  Ratio  Calculation 

•  Determine  the  SE/PM  cost.  Identify  the  line  item  for  “Systems 
Engineering/Program  Management”  in  the  column  for  “WBS  Reporting 
Elements”  (Column  B).  Record  the  “Total”  estimate  at  completion  cost  for 
Systems  Engineering/Program  Management  (Column  J). 

•  Determine  the  EAC  total  cost.  Identify  the  “Total  Cost  (Less  Profit  or 
Fee)”  in  the  column  for  “WBS  Reporting  Elements”  (Column  B).  Record 
the  “Total”  estimate  at  completion  cost  for  EAC  total  cost  (Column  J). 

•  Determine  the  SE/PM  cost  ratio.  Divide  the  SE/PM  cost  (Step  1)  by  the 
EAC  total  cost  (Step  2).  Record  the  SE/PM  cost  ratio  value. 
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D.  PROGRAM  LIST  AS  REPORTED  IN  SELECTED  ACQUISTION 
REPORT  (SAR) 

Active  Electronically  Scanned  Array  (AESA)  Radar 
AIM-9X/Short  Range  Air-to-Air  Missile 

AIM- 120  Advanced  Medium  Range  Air-to-Air  Missile  (AMRAAM) 

Airborne  and  Maritime/Fixed  Station  Joint  Tactical  Radio  System  (AMF  JTRS) 
AN/WQR-3,  Advanced  Deployable  System  (ADS) 

Apache  Block  IIIA  Remanufacture  (AB3A  REMANUFACTURE) 

AV-8B/Attack,  V/STOL,  Close  Air  Support  (Harrier  11+  Remanufacture) 

B-2  Radar  Modernization  Program 

Cobra  Judy  Replacement  (Cobra  Judy  Replacement) 

EA- 1 8G  Growler  (EA- 1 8G) 

Expeditionary  Fighting  Vehicle  (EFV) 

E-3  AWACS  Radar  System  Improvement  Program  (RSIP) 

E-2C  Reproduction 

Family  of  Advanced  Beyond  Line-of-Sight  Tenninals  (FAB-T) 

Family  of  Medium  Tactical  Vehicles  (FMTV) 

Guided  Multiple  Launch  Rocket  System/DPICM/Unitary/Alternative  Warhead 
(GMLRS/GMLRS  AW) 

Joint  Common  Missile  (JCM) 

Joint  Tactical  Radio  System  Ground  Mobile  Radio  (formerly  Cluster  1) 

(JTRS  GMR) 

Longbow  Hellfire  -  subsystem  of  the  AH-64  Apache  Weapon  System 
LHA  Replacement  Amphibious  Assault  Ship 

MQ-4C  Unmanned  Aircraft  System  Broad  Area  Maritime  Surveillance  (MQ-4C 
UAS  BAMS) 

Multi-Platform  Radar  Technology  Insertion  Program  (MP-RTIP) 

National  Polar-orbiting  Operational  Environmental  Satellite  System  (NPOESS) 
Presidential  Helicopter  Replacement  (VH-71)  Program 
P-8A  Poseidon 

Sense  and  Destroy  Armor  (SAD ARM) 

Small  Diameter  Bomb  Increment  II  (SDB  II) 

Space  Based  Infrared  System  (SBIRS)  High  Program 
Standard  Missile  (SM)  -  2  Block  IV 
Stryker  Family  of  Vehicles  (STRYKER) 

UH-72A  Light  Utility  Helicopter  (LUH) 

Warfighter  Information  Network  -  Tactical  (WIN-T) 
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